Saturday, 28 July 2007

Microsoft, MySQL and the skewed WAMP

In his OSCON keynote last Thursday (26 July 2007), Bill Hilf of Microsoft described his perspective of the "two steps forward, one step backward" relationship between Microsoft and the world of Open Source. Amongst the steps forward, I count the establishment of a web site microsoft.com/opensource, releasing some Microsoft software under an OSI compliant license, and -- specifically to MySQL -- the mentioning of the MySQL Connector for Visual Studio in his OSCON presentation.



Microsoft seems to be acknowledging that there is a heterogeneous world out there. It's not "all proprietary" versus "all Open Source". The community mixes and matches at will.

Many might think there are few commonalities between MySQL and Microsoft. But there are more than you'd think, and they're not limited to the common first letter of the alphabet.

The biggest commonality is WAMP. A simple real-world observation is that plenty of the organisations deploying LAMP applications, have developed them in whole or in part under WAMP.

MySQL doesn't have a platform agenda, so we want to make WAMP easy to use. If WAMP turns out to be good enough not just for development, but also for deployment, great! And there MySQL and Microsoft have a common interest: Making WAMP attractive enough for production deployments, as opposed to being a mere playing ground for LAMP as The Real Thing.

So while you can expect MySQL to continue to want to excel as the M in LAMP, and while I suspect it's a safe bet to expect Microsoft to continue to have a platform agenda, I don't think anybody should expect a dogmatic, black and white, all-or-nothing world. In reality, customers and users mix and match, and it's about time we all acknowledge that.

Bill Hilf made two points in his OSCON keynote that I'd like to comment upon as for how they apply to MySQL. In both cases, I'll conclude with a plea to contact me, if the thinking applies to you.

First, he pointed out the order-of-magnitude technical improvements that could very simply be made to the interface between IIS and PHP, once key IIS developers were connecting with key PHP developers. Such opportunities for improvements exist only if the products have been living in complete isolation from each other. And that's not the case with MySQL and Windows, which have peacefully coexisted since last century. This means that the technical fruits to be picked when making WAMP more attractive won't hang quite that low. But we do know that there are technical hurdles to overcome, and documentation to be improved, specifically when it relates to deployment of WAMP applications. Should you be one of the casualties thereof, I'd love to hear from you about how we should make WAMP more attractive. Email me at firstname@mysql.com!

Second, he pointed to the fear of the unknown, to the extent Linux is unknown to the Microsoft user base. "Can I go visit a Linux website?" was one of the questions he had been asked, not tongue-in-cheek, but for real. My take on this is that there is a perceived divide "Microsoft vs. Open Source" bigger than the divide out in the real world of IT. And I don't think it's in the interest of the user base to be lead to believe that solutions have to be fundamentalist. Paradoxically, I don't even think it's in Microsoft's interest. I think the world's perception of MySQL's user base may be skewed towards LAMP rather than WAMP, not just because L, A, M and P are all Open Source as opposed to W, but also because of a relative lack of acknowledgement for WAMP as a development (and deployment) stack. Should you have deployed large WAMP apps, I'd love to hear from you about WAMP specific success stories. Email me at firstname@mysql.com!

I can see history repeating itself in strange ways. In the early days, many picked MySQL "for development and testing only", fully convinced to deploy on other databases (and that was definitely the case with myself in the 1990s). But the "testing" phase dragged on and on, and in many cases MySQL turned out to be "good enough" for deployment.

That's how MySQL grew.

With proper attention from MySQL, Microsoft and our respective communities, perhaps WAMP in a growing number of scenarios can enjoy the same benefit of being "good enough for deployment"?

Friday, 27 July 2007

OSCON Lightning Talk: "State of the Dolphin"

Today at OSCON, MySQL co-founder Michael "Monty" Widenius and I presented the "State of the Dolphin" lightning talk.



My slides for this preso weren't too graphic, which makes them all the easier to reuse in this blog:

Use our new software!



  • Use MySQL 5.1, it‘s soon going RC

  • Use Falcon, it‘s soon going Beta (new transactional storage engine, faster than InnoDB on large servers)

  • Use MySQL Workbench (ER Tool), Now Beta

  • Use MySQL Proxy, just released

  • PHPers: Use mysqlnd (Native Driver)



Go test MySQL 5.1!



  • We‘re happy with the quality

    • More stable than 5.0 was four months after GA

    • RC happening very soon, GA within a few versions after that

    • A better MySQL 5.0 (thousands of small fixes)

  • We‘re happy with the new functionality

    • Table / Index Partitioning

    • Row-based replication – Transfers data instead of commands

    • Full-text indexing parser plugins – Flexible full text search

    • Disk-based Data Support for MySQL Cluster

    • Replication Support for MySQL Cluster

    • XPath Support - helps any customer wanting to better navigate and search XML documents stored in MySQL

    • Internal Task Scheduler (Events)




Other goodies coming soon (5.2, 6.0, ...)



  • Global Backup API

  • Falcon and Maria (MyISAM++) storage engine

  • Further new storage engines

  • Hash & Merge joins (faster subselects)

  • Federated tables over ODBC

  • Foreign key support for all engines




Participate in our Development!



  • Report bugs! Test them! Submit patches!

  • Hang out on Freenode IRC #mysql-dev

  • Subscribe to commits@lists.mysql.com to see our code reviews

  • Attend MySQL University, the foremost education for MySQL developers (of C/C++ code, not apps)

  • MySQL Forge: List your MySQL apps! Upload your code snippets! Fix the missing documents!

  • Go visit MySQL Forge Worklog

    • Voting for best features starting soon



Hiring!



We are hiring outstanding C/C++ developers with systems or database engineering experience.
Visit www.mysql.com/jobs or email resume to jpugh@mysql.com

Clash of the DB Egos: Seen the show? Read the article!

A good three months ago, I had the pleasure of being the referee for seven DB egos. Now, the match has appeared as a Dev Zone article:

One of the most popular keynotes of the MySQL Conference & Expo 2007 was called "The Clash of the DB Egos". It was a fight amongst seven database luminaries, all playing an important role either within MySQL AB or as providers of Storage Engines that work closely with MySQL. This article attempts at giving a picture of what the fight was about, through reciting the egos and the questions posed to them by the referee.

Saturday, 7 July 2007

OSCON Thu 26.7.2007: "The MySQL Architecture of Participation"

Coming to OSCON at the end of the month?



If so, I'd be happy to see you on Thursday 26 July 2007 5:20pm - 6:05pm in F150, where I will be talking about The MySQL Architecture of Participation.

Quoting from the O'Reilly session page:

While MySQL has been FOSS from the start and GPL since 2000, the development process at MySQL has been fairly concentrated around the company itself. One of the corporate goals for 2007 is to open up our entire development model:


  • ensuring that our already-opened tools are open enough (our bugs system at bugs.mysql.com, and our source control system with BitKeeper)

  • opening Worklog, our detailed roadmap and specifications, for commenting on MySQL Forge

  • opening internal documentation from our closed Wiki onto MySQL Forge Wiki

  • opening our internal chat sessions, moving them from our closed IRC server to Freenode



which all share the commonality of lowering the threshold for external developers to contribute to MySQL.

In this session, Arnö shares the experiences so far, concentrating on building up the network of connections between internal and external developers.

Heidelberg Dev Mtg for Community: Thu-Fri 20-21 Sep 2007

As noted already in March, we have decided to open up the MySQL Developer Meeting for selected members of the MySQL User Community, i.e. for MySQL users who have a need of interacting with our developers. In Sorrento 2006, Prag 2005 or Malta 2004, we had similar developer meetings -- but the external representation amounted to one (1) customer presentation.

In Heidelberg (one bus shuttle hour from Frankfurt airport), we'll do things differently.

Some key points:


  1. Community Days are concentrated to Thursday 20.9.2007 and Friday 21.9.2007: While we have very few sessions closed for the community even on the other days (Wed 21.9, Sat 22.9, Mon 24.9), our scheduling started from the insight that few community members can afford to stay for too many days. So meetings that are of most relevance for Community generally take place on Thu and Fri.

  2. MySQL University sessions are scheduled for the same Thursday and Friday. That's where we spread our accumulated knowledge on how to develop MySQL internally amongst those who code MySQL, the Connectors, and Tools. Invited community members are welcome to attend physically, and anyone can participate virtually at no cost.

  3. Team Exhibitions are a new form of gathering. MySQL engineering teams invite their fellow teams to celebrate success by showing what they themselves think is cool stuff -- already in production, or in beta, or just in alpha. This doesn't have to be flashy user interfaces. For instance, the Optimiser team may choose to demo a query that runs 75 % faster in 5.1 than in 5.0. We'll make sure that our non-attending Community members gets to hear about this through MySQL Forge. And naturally, any external community members are most welcome to attend the Team Exhibitions first hand, both to get an impression of what's happening, and to give us feedback.

  4. Every attending non-MySQLer will get two hosts: one Community Team Host and one Engineering Team Host. The Community Team Host is either Jay, Lenz, Colin, Giuseppe, David or myself (Lenz will tell you know who is your host), and we make sure you know where and when the meetings are, we introduce you to the proper Engineering team members, and try to ensure your expectations are met. The Engineering Team Host is whichever MySQL developer who is the domain expert on your topic. Meeting with this developer is probably the reason why you came to Heidelberg in the first place, but do take into account that such devs are likely to be busy with many meetings in Heidelberg.

  5. The detailed agenda is still under works. During the course of July, we expect to be able to publish the MySQL University schedule, as well as the tentative Team Exhibition schedule. Individual community members should also get a picture of what meetings they are likely to be interested in, through interaction at least with their Community Team hosts and in some cases even with their Engineering Team hosts.



External participants should expect to be firming up their travel schedule within the next week or two. This involves getting a firm booking from the Heidelberg Marriott Hotel (along Neckar River at Vangerowstrasse, a short distance from Heidelberg's Old Town), which is both our meeting venue and the place where nearly all MySQLers stay. Expect to be contacted by Lenz Grimmer, our Community Manager EMEA (or contact him yourself at firstname@mysql.com, if you want to be proactive). Lenz is keeping a list of the arrival and departure dates and times of our guests.

mysqlnd (the MySQL native driver for PHP) needs testers and benchmarkers

MySQL welcomes external verification by high-volume LAMP websites of the performance improvements gained by replacing libmysql with mysqlnd.

To recap some basics from the mysqlnd download page at http://dev.mysql.com/downloads/connector/php-mysqlnd/:


The MySQL native driver for PHP is an additional, alternative way to connect from PHP 5 and PHP 6 to the MySQL Server 4.1 or newer. It is a replacement for the libmysql, the MySQL Client Library. From now on you can use ext/mysqli either together with libmysql as you did in the past or with mysqlnd.

We have no plans to remove libmysql support from ext/mysqli, which would break existing applications. We just add a new, superior alternative to our PHP offerings.

For development discussion and bug reporting a new mailing list has been created php@lists.mysql.com. See also http://lists.mysql.com/.


Georg Richter has now sent the PHP extensions for mysqlnd to the PHP Core Team for reviewing. The interfaces to PHP may then be put directly into the PHP CVS repository, whereas mysqlnd remains to be downloaded from MySQL's download site.

The code base of mysqlnd has been synchronised with the PHP CVS repository of php.net. While mysqlnd was originally designed for PHP 6, Andrey Hristov has now finished the backport of mysqlnd to PHP 5 and added support for ext/mysql. Given that 60 % of the PHP user base is still on PHP 4, some heavy PHP users would like to see the backporting done all the way to PHP 4. We are evaluating that, but have no immediate plans for such a backport.

Our next ambition is to prove that mysqlnd is stable and meets the need of the most demanding PHP users. And that is where we need help from the user community. We are looking for testers among PHP users with a high server load, ideally willing to do benchmarks.

Typically, mysqlnd is at least as fast as libmysql. We are writing a MySQL Technical Whitepaper with further details on the performance of mysqlnd, which will be published soon. The document confirms previously published results for the Dell DVD Store, which show ext/mysqli with mysqlnd to be about 5% faster than libmysql. Now, we would like an external verification of our performance improvements, but a small web site is not proof enough, as differences will show up only under high load. We are curious to learn how big the overall speed-up is for a top web site, given that the time spent in the database interface is only a fraction of the overall web page generation time.

Our team has increased their Quality Assurance efforts to further stabilise mysqlnd. Code Coverage analysis with GCOV shows that our tests cover 88% of the underlying ext/mysql C-code, 93% of ext/mysqli and 83% of mysqlnd. We are proud of our improved figures. If you want to compare the values to those of PHP 4.4 and 5.2, check out http://gcov.php.net/.

Due to the help of Kent Boortz in the MySQL Build Team, we are building mysqlnd routinely on 28 platforms. We have run our test suite against more than 20 MySQL Server versions. Still, the positive lab tests need to be confirmed by field usage, quite as the performance figures.

So if you run a high load web site based on PHP and MySQL,

(i) read up on how to get mysqlnd, how to install it, and on the features & limitations as well as the FAQ on the download page

(ii) subscribe to php@lists.mysql.com at http://lists.mysql.com/

(iii) talk to Jay Pipes (at firstname@mysql.com) or Ulf Wendel (at firstname.lastname@mysql.com)! Or, better still, catch them on IRC (#mysql-dev on Freenode).