This post is in reply to Hadi’s post. Please go ahead and read it.

Done? Great, so let me try to respond, this time, from the point of view of someone who regularly asks for patches / pull requests.

Here are a few examples.

To make things more interesting, the project that I am talking about now is RavenDB which is both Open Source and commercial. Hadi says:

Numerous times I’ve seen reactions from OSS developers, contributors or merely a simple passer by, responding to a complaint with: submit a patch or well if you can do better, write your own framework. In other words, put up or shut up.

Hadi then goes on to explain exactly why this is a high barrier for most users.

• You need to familiarize yourself with the codebase.
• You need to understand the source control system that is used and how to send a patch / pull request.

And I would fully agree with Hadi that those are stumbling blocks. I can’t speak for other people, but in our case, that is the intention.

Nitpicker corner here: I am speaking explicitly and only about features here. Bugs gets fixed by us (unless the user already submitted a fix as well).

Put simply, there is an issue of priorities here. We have a certain direction for the project that we want to take it. And in many cases, users want things that are out of scope for us for the foreseeable future. Our options then become:

• Sorry, ain’t going to happen.
• Sure, we will push aside all the work that we intended to do to do your thing.
• No problem, we added that to the queue, expect it in 6 – 9 months, if we will still consider it important then.

None of which is an acceptable answer from our point of view.

Case in point, facets support in RavenDB was something that was requested a few times. We never did it because it was out of scope for our plan, RavenDB is a database server, not a search server and we weren’t really sure how complex this would be and how to implement this. Basically, this was an expensive feature that wasn’t in the major feature set that we wanted. The answer that we gave people is “send me a pull request for that”.

To be clear, this is basically an opportunity to affect the direction of the project in a way you consider important. What ended up happening is that Matt Warren took up the task and created an initial implementation. Which was then subject to intense refactoring and finally got into the product. You can see the entire conversation about this here. The major difference along the way is that Matt did all the research for this feature, and he had working code. From there the balance change. It was no longer an issue of expensive research and figuring out how to do it. It was an issue of having working code and refactoring it so it matched the rest of the RavenDB codebase. That wasn’t expensive, and we got a new feature in.

Here is another story, a case where I flat out didn’t think it was possible. About two years ago Rob Ashton had a feature suggestion (ad hoc queries with RavenDB). Frankly, I thought that this was simply not possible, and after a bit of back and forth, I told Rob:

Let me rephrase that.
Dream up the API from the client side to do this.

Rob went away for a few hours, and then came back with a working code sample. I had to pick my jaw off the floor using both hands. That feature got a lot of priority right away, and is a feature that I routinely brag about when talking about RavenDB.

But let me come back again to the common case, a user request something that isn’t in the project plan. Now, remember, requests are cheap. From the point of view of the user, it doesn’t cost anything to request a feature. From the point of view of the project, it can cost a lot. There is research, implementation, debugging, backward compatibility, testing and continuous support associated with just about any feature you care to name.

And our options whenever a user make a request that is out of line for the project plan are:

• Sorry, ain’t going to happen.
• Sure, we will push aside all the work that we intended to do to do your thing.
• No problem, we added that to the queue, expect it in 6 – 9 months, if we will still consider it important then.

Or, we can also say:

• We don’t have the resources to currently do that, but we would gladly accept a pull request to do so.

And that point, the user is faced with a choice. He can either:

• Oh, well, it isn’t important to me.
• Oh, it is important to me so I have better do that.

In other words, it shift the prioritization to the user, based on how important that feature is.

We recently got a feature request to support something like this:

session.Query<User>()
   .Where(x=> searchInput.Name != null && x.User == searcInput.Name)
   .ToArray();

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

I’ll spare you the details of just how complex it is to implement something like that (especially when it can also be things like: (searchInput.Age > 18). But the simple work around for that is:

var q = session.Query<User>();
if(searchInput.Name != null)
  q = q.Where(x=> x.User == searcInput.Name);

q.ToArray();

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

Supporting the first one is complex, there is a simple work around that the user can use (and I like the second option from the point of view of readability as well).

That sort of thing get a “A pull request for this feature would be appreciated”. Because the alternative to that is to slam the door in the user’s face.

Several years ago, I worked for a company where I felt my potential wasn’t being tapped into. I didn’t feel challenged and quite frankly, I felt underemployed. My husband had mentioned to me that he knew about a company, by the name of Pragmatic Works, that gave away a free week of I.T. training to people who were interested in getting into a new career of Business Intelligence. To be honest, I didn’t know what that meant exactly, but it had already sounded better than what I was currently doing. You had to meet certain criteria to even apply: military veterans, unemployed or underemployed.

After submitting the required information, I was accepted into the Foundation. Now, I figured it would be incredibly challenging considering that extent of my experience with computers consisted of checking email and using company software. It took me 1 hour to drive to this little office out in Green Cove Springs. When I showed up on the first day of training, there were about seven other people standing around this little conference room table. You could see that some of the students felt just as apprehensive as I did. We really had no clue what were getting ourselves into.

Our instructor, Devin Knight, whom I might add was best suited for all of us newbie’s, patiently waited for us to get settled in. Devin covered the agenda for the week, which was around T-SQL and Reporting Services. To be honest with you, I think that it could have been the first time most of us heard those words. As the morning progressed, I realized that this would be a challenging week. During a break in the class, I spoke to my husband about booking a hotel for the week so I didn’t spend 2 hours a day on the road traveling back and forth to the office. I wanted to use that time to focus on studying and really just throwing myself into this great opportunity.

Even though I was just scratching the surface in my knowledge for TSQL and Reporting Services, I felt pretty good about what I had learned and I definitely felt like it would be a career path that would consistently push me forward and not allow me to become complacent like other positions I’ve held. Shortly after the Foundation class ended for me, I interviewed for a junior position within Pragmatic Works and received the position. Now let me fast forward a bit.

In my position, I was growing rapidly. A lot of it had to do with an enormous amount of late night studying and really just committing myself. I focused on growing my TSQL and Reporting Services skills and by the time the company was ready to hold another Foundation class, I asked if I could teach it. I felt so blessed and honored to have received that opportunity and it made such a huge difference to me personally and professionally, that I wanted to be a testimonial for other people who were also feeling the way I felt when I first arrived to the Foundation.

On the very first day of the Foundation class that I taught, as I told my story, all the students were shocked that in less than a year, I had sat on the other side of that table as a student myself. After the class ended that week, I felt such a high. I was fortunate enough to be in a position to share my experience and what some of the possibilities are with commitment, drive, passion and enthusiasm.

We are now at our present date and time. The Foundation has been going strong for 3 years now. The demand how grown so much that we went from teaching the class, 2 times a year, to quarterly, to bi-monthly and now monthly. After the conclusion of the weeklong training, students have the option of doing an internship to receive additional training. We have had over 100 students participate in the Foundation and over half of those students have found full time work in either a DBA or BI role.

The Foundation still sticks by its core values:

As individuals and as a company, we have received many great, unexpected opportunities. It is our responsibility socially, to pay that forward to individuals who might not expect to receive those very same equal opportunities. For the men and women who have served our country, you will have first rights to our training opportunities, for we cannot say thank you enough for the daily sacrifice that you are making.

For more information about the Pragmatic Works Foundation, please visit http://pragmaticworks.org.

Sherri McDonald
Business Development Manager
Pragmatic Works

(Posted February 7, 2012) That’s right.  If you’ve been keeping score you know that we had already announced End of Marketing (EoM) for DB2 9 last year.   That happens on December 10, 2012.   When I discussed that date in this blog I did mention that ...

Many developers have contacted us in the past months asking for a free version of Xeround’s database-as-a-service — and we’re happy to oblige!

We’re excited to announce the much anticipated release of “Xeround FREE” - a free version of our popular cloud database. Xeround FREE provides MySQL users with a zero-management, simple, highly available – and best of all completely free - cloud database solution.

CHILL with Your Database in the Cloud

Rather than having to install, configure and maintain a MySQL database in the cloud, you can now easily create your free cloud database with just one click.

With database creation and management out of the way — you can focus on your code rather than on the IT side of things.

Xeround FREE is great for starting out with your database in the cloud. It is suitable for on-going development, testing and staging purposes, as well as for smaller web applications. Available up to 10MB at no charge, Xeround FREE is initially offered on our Amazon EC2 US-East datacenter and as an Heroku add-on, expanding to additional regions/clouds in the future.

Free as in Beer

Xeround FREE is completely free. Forever.

Every Xeround user is entitled to one concurrent Xeround FREE database instance.

To create your free cloud database, simply log in to your account and choose “Create New”, selecting “Xeround FREE” from the list of available editions.

Xeround Cloud Database New Editions

For customers that require larger database sizes, higher levels of scalability, additional support and SLA, we are announcing two additional editions of our cloud database solution: Xeround BASIC and Xeround PRO.

Xeround BASIC is suitable for medium-size applications looking for a cost-effective, plug-&-play, highly available database solution in the cloud. Xeround BASIC is limited up to 250MB and is offered in tiered pricing starting at $17/month.

Xeround PRO (our commercial version that launched last June) is suitable for mission-critical, larger-scale applications that require unlimited scalability, high availability SLA, scheduled backups and more. Xeround PRO already has more than 300 paying customers and boasts a complete end-to-end solution for seamlessly running MySQL applications in the cloud. Xeround PRO is offered in a pay-per-use model starting at $0.21 for GB/hour of data (with a minimum charge for 0.5GB), and $0.43 for GB of data transfer.

Both Xeround BASIC and Xeround PRO are available on Xeround’s datacenters on Amazon EC2 in the US and in Europe, Rackspace and on the Heroku platform.

The first 30 days of using Xeround BASIC or Xeround PRO are free!

To start your free trial register to Xeround, log in to your account and choose “Create New” – selecting either “Xeround BASIC” or “Xeround PRO” from the list of available plans. The 30-day free trial is in addition to your Xeround FREE database instance.

Xeround’s new editions provide you the flexibility to choose the most suitable plan for your needs.

To learn more on the differences between Xeround FREE, Xeround BASIC and Xeround PRO, visits our Plans and Pricing page.

Xeround customers: Check your email for instructions on how to migrate your existing DB to any one of the three new editions.
 

You may also be interested in:

• Connecting Your MySQL-Heroku Application to Xeround Cloud Database


• Cloud PaaS — Application Scalability, But What About Database Scalability?


• Xeround Cloud Database on Rackspace Now Open for Business!

The latest release of Xeround Cloud Database out today now supports round-robin DNS on all of Xeround’s datacenters.

Round-robin DNS guarantees high availability of the connection to your database in a way that does not require you to tweak your application.

In the Database Manager Online Console you can see the DNS hostnames for connecting to your database.

• The Internal DNS points to the internal IP addresses and is used for accessing your DB from within the same datacenter
• The External DNS is used for connecting externally.

Each DNS routes to one of the available IP addresses we supply for connecting to your database. That way, if one IP address is temporarily unavailable, you can still connect to your database instance.

Under the Hood:

Xeround Cloud Database has multiple – and unlimited – front ends that can scale out as the application demands. We supply each DB instance with several IP addresses that are in fact load balancers that route the SQL requests to any of the database’s multiple front- ends. Those IP addresses are being resolved by the round-robin DNS.

What could cause a connection to fail?

Say a cloud server crashes (happens!). Xeround’s service management layer will monitor the machine and will remove it from the DNS entry so further attempts will not try to connect to the failed server. Moreover, Xeround’s auto-healing mechanism employs a “stand-by” machine, which will take over the failed components and then be added to the DNS entry, so further connections could be diverted to it. In the past, you had to code the failover between the connection IPs yourself, whereas now the round-robin DNS simplifies high-availability of your database connections in a way that’s completely transparent to the application.

What does the new DNS mean for our current customers:

New database instances that are created get the round-robin DNS by default. 

Existing customers or trial users that wish to take advantage of the DNS and other improvements of the new release, need to log in to their account and choose “Change Plan” to switch to one of the new editions. Switching to the new version will assign new connection DNS hostnames to your database.

For instructions and code samples for connecting your application to Xeround Cloud Database see here.

If you have any questions or need assistance, don’t hesitate to contact us.

You may also be interested in:

• Delivering High Availability for MySQL Apps on the Cloud


• Connecting Your MySQL-Heroku Application to Xeround Cloud Database


• Xeround Cloud Database Secures $9 Million in Funding Round

In my previous post, I explained why I really don’t the following.

public class CargoInspectionServiceImpl : ICargoInspectionService 
{
  // code redacted for simplicity

 public override void InspectCargo(TrackingId trackingId)
 {
    Validate.NotNull(trackingId, "Tracking ID is required");

    Cargo cargo = cargoRepository.Find(trackingId);
    if (cargo == null)
    {
      logger.Warn("Can't inspect non-existing cargo " + trackingId);
      return;
    }

    HandlingHistory handlingHistory = handlingEventRepository.LookupHandlingHistoryOfCargo(trackingId);

    cargo.DeriveDeliveryProgress(handlingHistory);

    if (cargo.Delivery.Misdirected)
    {
      applicationEvents.CargoWasMisdirected(cargo);
    }

    if (cargo.Delivery.UnloadedAtDestination)
    {
      applicationEvents.CargoHasArrived(cargo);
    }

    cargoRepository.Store(cargo);
 }
}

Now, let us see one proposed solution for that. We can drop the IApplicationEvents.CargoHasArrived and IApplicationEvents.CargoWasMisdirected, instead creating the following:

public interface IHappenOnCargoInspection
{
   void Inspect(Cargo cargo);
}

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

We can have multiple implementations of this interface, such as this one:

public class MidirectedCargo : IHappenOnCargoInspection
{
   public void Inspect(Cargo cargo)
   {
        if(cargo.Delivery.Misdirected == false)
             return;

        // code to handle misdirected cargo.
   }
}

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

In a similar fashion, we would have a CargoArrived implementation, and the ICargoInspectionService would be tasked with managing the implementation of IHappenOnCargoInspection, probably through a container. Although I would probably replace it with something like:

container.ExecuteOnAll<IHappenOnCargoInspection>(i=>i.Inspect(cargo));

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

All in all, it is a simple method, but it means that now the responsibility to detect and act is centralized in each cargo inspector implementation. If the detection of misdirected cargo is changed, we know that there is just one place to make that change. If we need a new behavior, for example, for late cargo, we can do that by introducing a new class, which implement the interface. That gives us the Open Closed Principle.

This is better, but I still don’t like it. There are better methods than that, but we will discuss them in another post.

(Posted February 7, 2012) That’s the question. However, it’s not my question. It’s the question that a friend of mine, Martin Packer, put to his readers in his blog today.   It’s a legitimate question and its a question of interest to me as well. In Martins blog...

We’ve released Percona Toolkit 2.0.3, with a couple of major improvements and many minor ones. You can download it, read the documentation, and get support for it. What’s new? You can read the changelog for the details, but here are the highlights: Brand new pt-diskstats, thanks to Brian Fraser. This tool is completely rewritten, and [...]

LevelDB: SSTable and Log Structured Storage:

Ilya Grigorik digs into LevelDB’s SSTable and log structured storage1:

If Protocol Buffers is the lingua franca of individual data record at Google, then the Sorted String Table (SSTable) is one of the most popular outputs for storing, processing, and exchanging datasets. As the name itself implies, an SSTable is a simple abstraction to efficiently store large numbers of key-value pairs while optimizing for high throughput, sequential read/write workloads.

Even if not very talked about, LevelDB is making notable contributions to the NoSQL space: the leveled compaction strategy in Cassandra 1.0 is based on LevelDB and Riak ships with LevelDB since 1.0.

1. Make sure you are not missing Writes Performance: B+Tree, LSM Tree, Fractal Tree ↩

Original title and link: LevelDB: SSTable and Log Structured Storage (NoSQL database©myNoSQL)

By pure coincidence, General Chicken just published on High Scalability a bullet point summary of the 99designs architecture I’ve linked and commented on earlier.

Original title and link: The Design of 99designs - A Clean Tens of Millions Pageviews Architecture (NoSQL database©myNoSQL)

The SQL Server 2012 Virtual Launch Event kicks off on March 7, 2012 starting at 8:00AM Pacific (-8:00 GMT) with keynotes from Microsoft Executives including: Ted Kummert, Corporate Vice President, Business Platform Division and Quentin Clark, Corporate Vice President, Database Systems Group. Following the keynotes, you’ll have access to over 30+ sessions to learn about the new capabilities of SQL Server 2012 at your own pace, on your own schedule.

The following sessions below are recommended for all DBAs and IT Professionals that want to learn all about the exciting New World of Data with SQL Server 2012. During the first 24 hours of the virtual launch event, you’ll be able to live chat with members of the SQL Server team to help answer your questions.

Register today at www.sqlserverlaunch.com for the SQL Server 2012 Virtual Launch Event and start adding the recommended sessions below to your calendars.

Breakthrough Insights Sessions:

Rapid Data Exploration: Powerful Self-Service Analysis with PowerPivot
Get a taste of what's new in PowerPivot, the self-service BI experience that enables information workers to build their personal and team BI applications within the familiar environment of Excel: This session will provide a quick introduction to PowerPivot, followed by demos highlighting the top 5 new modeling features delivered with the release of SQL Server 2012.
Duration: 20 Minutes
Speaker: Julie Strauss, Senior Program Manager, Microsoft   

Credible, Consistent Data: Improving Data Quality with the New Data Quality Services
No matter how brilliant your BI tools are, it's garbage-in-garbage-out. This session will give an overview on the new Data Quality Services product in SQL Server 2012. Come watch this important scenario to learn how to position, sell, and demo the new DQS product.
Duration: 20 Minutes
Speaker: Ashvini Sharma, Principal Program Manager Lead, Microsoft

Credible, Consistent Data: Improving Data Quality with the New Data Quality Services
No matter how brilliant your BI tools are, it's garbage-in-garbage-out. This session will give an overview on the new Data Quality Services product in SQL Server 2012. Come watch this important scenario to learn how to position, sell, and demo the new DQS product.
Duration: 20 Minutes
Speaker: Gadi Peleg, Senior Program Manager, Microsoft

Why Should Oracle DBAs Consider Microsoft SQL Server
Today’s medium and large enterprises are complex entities, built on a wide range of technologies. The biggest challenge emerging for IT implementers is cross-platform database management. In a recent survey on DB trends, a third of the respondents said they expect more DB platforms to be introduced in their organizations next year. Learn why Oracle DBAs should consider becoming bilingual DBAs and how Microsoft can help them grow and advance their IT careers.
Duration: 20 Minutes
Speaker: Bill Ramos, SQL Server Workstream Manager, Advaiya Inc

Mission Critical Confidence Sessions:

Required 9s and Data Protection: Introduction to SQL Server 2012 AlwaysOn, New High Availability Solution
This session provides an overview of AlwaysOn and explains how you can build a high availability solution using the new features and capabilities. You will learn how to deploy an AlwaysOn solution, how to utilize your secondary hardware for better return of investment and how the new solution simplifies deployment, management of high availability in your environment. The session will also describe how to manage application connectivity of your primary OLTP applications and reporting applications in an AlwaysOn environment.
Duration: 20 Minutes
Speaker: Santosh Balasubramanian, Senior Program Manager, Microsoft

Required 9s and Data Protection: Protect Data with Higher Chance of Recovery - Includes Recovery Advisor
In this session we will look at new backup and restore capabilities in SQL Server 2012 in order to enhance database protection for customers.
Duration: 20 Minutes
Speaker: Kevin Farlee, Senior Program Manager, Microsoft

Blazing-Fast Performance: A Technical Best Practices Tour with ColumnStore Index
Go behind the scenes with SQL Server engineers for a quick technical tour of the ColumnStore Index feature and an introduction to best practices for achieving astonishing performance with your ColumnStore Indexes.
Duration: 20 Minutes
Speakers:
Susan Price, Senior Program Manager, Microsoft
Eric Hanson, Principal Program Manager Lead, Microsoft

Blazing-Fast Performance: Explore FileTables, Full-Text Search and Semantic Search for Faster, More Meaningful Results Across Unstructured Data
80% or more of the data produced and stored are so called unstructured documents and are often not stored in the database, but often need to be managed in conjunction with relational data. In order to facilitate the storage and management of unstructured data, SQL Server has evolved from storing blobs to providing a sophisticated integration with the Windows Filesystem and provides rich services over such data to provide an interesting and compelling application development experience. This presentation introduces the new unstructured data processing capabilities of SQL Server 2012 that provide full Windows application-compatible file management over files stored in SQL Server as well as the services to unlock the information in such data such as the extensions and improvements to Full-Text Search and the introduction of semantic similarity search.
Duration: 20 Minutes
Speakers:
Il-Sung Lee, Senior Program Manager, Microsoft
Jack Richins, Senior Developer Lead, Microsoft

Cloud on Your Terms Sessions:
Big Data Analytics with Apache Hadoop and SQL Server 2012
In this session we will explore Apache Hadoop on Windows Server and Azure and powerful insights we can generate integrating with SQL Server.
Duration: 20 Minutes
Speaker: Saptak Sen, Senior Product Technical Manager, Microsoft

Optimized Productivity: T-SQL Enhancements
SQL Server 2012 comes with an impressive addition of T-SQL functionality. These enhancements cover a range of topics such as ease-of-use, performance and improved metadata discovery. In this session Tobias will give an overview of these features and dive a bit deeper into some of them using live-coding demos.
Duration: 20 Minutes
Speaker: Tobias Ternstrom, Senior Program Manager Lead, Microsoft

Synchronizing Data “From On-Prem to Cloud” or “Cloud to Cloud” Using SQL Azure Data Sync
We’ll take a look at SQL Azure Data Sync and see how it can be used to provide an on-going data synchronization for various scenarios like hybrid applications (on-prem to cloud), scale out (cloud to cloud), geo distribution (cloud to cloud across data centers) and so on. We will also show you how SQL Azure Data Sync can be used with other Azure platform capabilities like Azure Reporting Services.
Duration: 20 Minutes
Speaker: Mark Scurrell, Senior Program Manager Lead, Microsoft

Scale on Demand: Optimizing SQL Server 2012 for Private Cloud
In this session we will cover key private cloud enabling features in SQL Server 2012.
Duration: 20 Minutes
Speaker: Xin Jin, Program Manager, Microsoft

Register today at www.sqlserverlaunch.com to meet the new Microsoft cloud-information platform, SQL Server 2012!

99designs: Powered by Amazon RDS, Redis, MongoDB, and Memcached:

While the authoritative storage is Amazon RDS, 99designs is using Redis, MongoDB, and Memcached for transient data:

We log errors and statistics to capped collections in MongoDB, providing us with more insight into our system’s performance. Redis captures per-user information about which features are enabled at any given time; it supports our development stragegy around dark launches, soft launches and incremental feature rollouts.

It’s also worth noting the nice things they say about using Amazon RDS:

An RDS instance configured to use multiple availability zones provides master-master replication, providing crucial redundancy for our DB layer. This feature has already saved our bacon multiple times: the fail over has been smooth enough that by the time we realised anything was wrong, another master was correctly serving requests. Its rolling backups provide a means of disaster recovery. We load-balance reads across multiple slaves as a means of maintaining performance as the load on our database increases.

Original title and link: 99designs: Powered by Amazon RDS, Redis, MongoDB, and Memcached (NoSQL database©myNoSQL)

A great post by Olivier Mallassi on a topic that comes up very often: how do data grids and NoSQL databases compare?

• Data Grids enable you controlling the way data is stored. They all have default implementation (Gigaspaces offers RDBMS by default, Gemfire offers file and disk based storage by default….) but in all cases, you can choose the one that fits your needs: do you need to store data, do you need to relieve the existing databases….
• In order to minimize the latency, data grids enable you to store data synchronously (write-through) or asynchronously (write-behind) on disk. You can also define overflow strategies. In that case, data is store in memory up to a treshold where data is flushed on disk (following algorithms like LRU …). NoSQL solutions have not been designed to provide these features.
• Data grids enable you developing Event Driven Architecture.
• Querying is maybe the point on which pure NoSQL solutions and data grids are merging.
• Data grids enable near-cache topologies.

Taking a step back you’ll notice that there are actually more similarities than differences. While Oliver Mallasi lists the above points as features that prove data grids as being more configurable and so more adaptable, some of these do exist also in the NoSQL databases taking different forms:

1. pluggable storage backends. Not many of the NoSQL databases have this feature,but Riak and Project Voldemort are offering different solutions that are optimized for specific scenarios.
2. replicated and durable writes. Not the same as synchronous vs asynchronous writes, but a different perspective on writes.
3. Notification mechanisms. Once again not all of the NoSQL databases support notification mechanisms, but a couple of them have offer some interesting approaches:
   1. CouchDB: _changes feed with filters
   2. Riak: pre-commit and post-commit hooks
   3. HBase coprocessors
4. Most of the NoSQL database have local per-node caches.

With these, I’ve probably made things even blurrier. But let me try to draw a line between data grids and NoSQL databases. Data grids are optimized for handling data in memory. Everything that spills over is secondary. On the other hand, NoSQL databases are for storing data. Thus they focus on how they organize data (on disk or in memory) and optimize access to it. Data grids are a processing/architectural model. NoSQL databases are storage solutions.

Original title and link: Data Grid or NoSQL? What are the common points? The main differences? (NoSQL database©myNoSQL)

On a Windows Phone 7 project I am currently building I had a need to build a factory that instantiates different concrete classes depending upon an object at runtime.   I have a number of information feeds, that all have different classes built to actually go get their data.  One for RSS Feeds, one for Twitter, etc.  Each Feed has a property for the FeedType, this is the information needed to determine what class to instantiate at runtime.

I have written this pattern many times, but I ran across an interesting post on StackOverflow where Steven gave a different approach to solving this without the classic giant switch statement in the Create method.   I thought it was neat enough to use in my application and share here.

There are a couple of steps you have to take, but none of them are difficult.

• Decorate your objects with a custom attribute
• Build the object types using reflection
• Implement your factory method

Custom Attribute

The attribute I added is called FeedTypeAttribute, I then add it to the concrete feed processor classes.

class FeedTypeAttribute : Attribute
{
    private int _feedType;

    public FeedTypeAttribute(int feedType)
    {
        _feedType = feedType;
    }

    public int FeedTypeId
    {
        get
        {
            return _feedType;
        }
    }
}

Adding the attribute to a class looks like the following.  

[FeedTypeAttribute(1)]
public class StaticFeedProcessor : IFeedProcessor
{
    public StaticFeedProcessor()
    {

    }

    public void ProcessFeedModel(FeedModel feedModel)
    {
        throw new NotImplementedException();
    }
}

Note that this class would be a FeedType of 1 in my data, but how do you determine what class that is during runtime?  The answer is reflection (and yes, this does work on Windows Phone 7).   Factory Method Using Custom Attributes

public class FeedProcessorFactory : IFeedProcessorFactory
{
    private static Dictionary<int, Type> processorList = new Dictionary<int, Type>();

    static FeedProcessorFactory()
    {
        // Get the types in this assembly that implement our custom attribute
        var targetTypes = from type in Assembly.GetExecutingAssembly().GetTypes() 
                         where type.CanBeTreatedAsType(typeof(IFeedProcessor))
                         where !type.IsAbstract && !type.IsInterface
                         let customAttributes = type.GetCustomAttributes( typeof(FeedTypeAttribute), false)
                         let attribute = customAttributes[0] as FeedTypeAttribute
                         select new { type, attribute.FeedTypeId };

        processorList = targetTypes.ToDictionary(p => p.FeedTypeId, p => p.type);
    }

    public IFeedProcessor CreateProcessorByFeedType(int feedId)
    {
        Type feedType = processorList[feedId];

        return Activator.CreateInstance(feedType) as IFeedProcessor;
    }
}

The key here is the type.GetCustomAttributes() call.  The linq expression gets all the type of the current assembly where the type can be assigned from the IFeedProcessor class.  We have to ensure the class is not abstract or an interface, and then get the custom attributes. 

The let clause in the linq expression can be thought of like a local temporary variable.  Assigning the attributes to it, and then processing them in the final select.

Consuming the Factory

Consuming the factory is then as simple as instantiating a factory object and calling the CreateProcessorByFeedType function.  A fully allocated processor is returned ready to handle the feed.

// This is in my container object
private static IFeedProcessorFactory _feedProcessorFactory = new FeedProcessorFactory();


// This is in the function to process the feeds
IFeedProcessor processor = _feedProcessorFactory.CreateProcessorByFeedType(feed.FeedType);
processor.ProcessFeedModel(feed);

Less Coupling is Good This code allows for the addition of new processor classes just by building the class and adding the custom attribute.  Of course you should check if the feedtype is not supported before trying to use the object (I omitted some code used for safety checking).   If a newer class is created in the future to replace a current one I only have to change the custom attribute to handle it.  No switch statements, or remembering to update a config somewhere.

Redis and Python: Building a Markov-chain IRC bot:

Charles Leifer:

As an IRC bot enthusiast and tinkerer, I would like to describe the most enduring and popular bot I’ve written, a markov-chain bot. Markov chains can be used to generate realistic text, and so are great fodder for IRC bots.

Redis acts, in many ways, like a big python dictionary that can store several types of useful data structures. For our purposes, we will use the set data type. The top-level keyspace will contain our “keys”, which will be encoded links in our markov chain. At each key there will be a set of words that have followed the words encoded in the key. To generate “random” messages, we’ll use the “SRANDMEMBER” command, which returns a random member from a set.

1. You could use other NoSQL database for this, but you’d miss Redis’s support for sets and the O(1) SRANDMEMBER
2. On the other hand imagine storing the corpus in a graph database where the nodes would represent the words and vertices would carry to pieces of information: the frequency of the connection in the corpus and the frequence of the connection used to generate the output.

Original title and link: Redis and Python: Building a Markov-chain IRC bot (NoSQL database©myNoSQL)

Calculating a Graph's Degree Distribution Using R MapReduce over Hadoop:

Marko Rodriguez is experimenting with R on Hadoop and one of his exercises is calculating a graph’s degree distribution. I confess I had to use Wikipedia for reminding what’s the definition of a node degree:

1. The degree of a node in a network (sometimes referred to incorrectly as the connectivity) is the number of connections or edges the node has to other nodes. The degree distribution P(k) of a network is then defined to be the fraction of nodes in the network with degree k.
2. The degree distribution is very important in studying both real networks, such as the Internet and social networks, and theoretical networks.

As an imagination exercise think of a graph database that’s actively maintaining an internal degree distribution and uses it to suggest or partition the graph. Would that work?

Original title and link: Calculating a Graph’s Degree Distribution Using R MapReduce over Hadoop (NoSQL database©myNoSQL)

Pierre Bailet and Mathieu Poumeyrol of fotopedia (a French photo site) share their experience of operating a small MongoDB cluster since Sep.2009 compared to a MySQL cluster.

Some details about fotopedia:

• fotopedia is 100% on AWS
• Amazon RDS for MySQL
• 4 nodes MongoDB cluster
• 150mil. photo views

MongoDB advantages:

• no alter table
• background index creation
• data backup & restoration
  • note: as far as I can tell MySQL is able to do the same
• replica sets
• hardware migration
  • note: the same procedure can be used for MySQL

Before leaving you with the slides, here is an interesting accepted trade-off:

Quietly losing seconds of writes is preferable to:

• weekly minutes-long maintenance periods
• minutes-long unscheduled downtime and manual failover in case of hardware failures

Original title and link: MongoDB vs MySQL: A DevOps point of view (NoSQL database©myNoSQL)

Whirr and Hadoop Quickstart Guide: Automating a Rackspace Hadoop Cluster:

Even if most of the examples show Whirr in action on the Amazon cloud, Whirr it’s cloud-neutral. Bob Gourley uses Whirr to fire up a CDH1 cluster on Rackspace.

1. Cloudera Distribution of Hadoop. ↩

Original title and link: Whirr and Hadoop Quickstart Guide: Automating a Rackspace Hadoop Cluster (NoSQL database©myNoSQL)

February is going to be a busy meetup month for me. In addition to speaking in Raleigh I will visit Charlotte to speak at Meetup out there on February 23rd. Last year I visited Charlotte meetup was in the great place and we had great food and great crowd of people showing up. I will [...]

In my previous post, I showed an example of application events and asked what is wrong with them.

public class CargoInspectionServiceImpl : ICargoInspectionService 
{
  // code redacted for simplicity

 public override void InspectCargo(TrackingId trackingId)
 {
    Validate.NotNull(trackingId, "Tracking ID is required");

    Cargo cargo = cargoRepository.Find(trackingId);
    if (cargo == null)
    {
      logger.Warn("Can't inspect non-existing cargo " + trackingId);
      return;
    }

    HandlingHistory handlingHistory = handlingEventRepository.LookupHandlingHistoryOfCargo(trackingId);

    cargo.DeriveDeliveryProgress(handlingHistory);

    if (cargo.Delivery.Misdirected)
    {
      applicationEvents.CargoWasMisdirected(cargo);
    }

    if (cargo.Delivery.UnloadedAtDestination)
    {
      applicationEvents.CargoHasArrived(cargo);
    }

    cargoRepository.Store(cargo);
 }
}

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /white-space: pre;/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

This is very problematic code, from my point of view, for several reasons. Look at how it allocate responsibilities. IApplicationEvents is supposed to execute the actual event, but deciding when to execute the event is left for the caller. I said several reasons, but this is the main one, all other flows from it.

What happen when the rules for invoking an event change? What happen when we want to add a new event?

The way this is handled is broken. It violates the Open Closed Principle, it violates the Single Responsibility Principle and it frankly annoys me.

Can you think about ways to improve this?

I’ll discuss some in my next post.