Wednesday, 28 March 2007

Third GPLv3 Draft Released

Today, FSF released the third GPLv3 draft on http://gplv3.fsf.org/gpl3-dd3-guide . It's great that the entire, complex process now has completed its next milestone, and I can fully understand why the FSF has seen it necessary to proceed at a pace somewhat slower than originally intended.

My thoughts go back well over a year to the Free Software Foundation's first conference on GPLv3, at the end of January 2006. It was a great meeting, and as a Committee B member and representative of MySQL, I was asked by China Martens of IDG for my input. My reply "I think there was a good balance between different hair and beard lengths" ended up on China's compilation "2006: The IT year in quotes" as "a colourful description of the cross-section of attendees".

I wish I had something equivalently colourful to describe the third draft of GPLv3. "A good balance between FSF's interest to protect software freedom in the name of the end user, and the interests of the industry to develop sustainable business models upon Free Software"? I hope I will feel able to use that for the final GPLv3 one day.

For now, MySQL continues to be actively engaged in the Free Software Foundation’s GPLv3 drafting process, the next step of which is the upcoming 60 days that should end up in a final draft. Our flagship product, the MySQL database server, remains licensed under GPLv2.

Until the new version of GPL is finalised, we won’t be in a position to determine whether GPLv3 is an appropriate license for MySQL products. As I said at the end of last year in my blog, until we get clear and strong indications for the general acceptance of GPLv3 over GPLv2, we feel comfortable with a specific GPLv2 reference in our license. So while we support and care for the GPLv3 process, don't expect us to be amongst the first GPLv3 movers.

By the way, don't miss Eben Moglen's keynote at the MySQL Conference & Expo 23-26 April 2007 in Santa Clara, California! Eben is not only a brilliant mind and shaper of the future, but one of the best public speakers I have ever heard.

Evaluating the MySQL Summer of Code Applications



Having taken a look at the 35 MySQL Summer of Code applications, I can now say that we have several very strong applications, such as:


  • IPv6: Enabling MySQL to connect over IPv6 internet

  • MySQL Based Atom Store (storing Google GDATA in MySQL)

  • Test case development / Load testing

  • Test suite development

  • MySQL Auditing Software

  • The "anti-profiler" - a performance analysis tool for database engines



For several of the above, we have multiple good applications. The applicants come from recognised universities across the world, not just the US and Europe. Some have extensive experience from professional software development. Sadly, we will end up having to reject several good proposals.

In addition to the challenging task of picking the best applications, we also have to prepare them for Google's final approval. We expect to get it done by Friday, and Google to have approved (a subset of) our recommendations by the end of next week.

Tuesday, 27 March 2007

35 Applications for MySQL Summer of Code Grants

Today, the time closed to apply for student positions at MySQL Summer of Code. We are very fortunate and thankful to have got 35 applicants fulfilling the formal requirements of an application fully entered and coupled with a task announced on http://forge.mysql.com/wiki/SummerOfCode -- thank you!



Our next task is to narrow this group down to the number of projects which can actually start. This will be a much smaller number. Even in the theoretical case that Google would allow us to have 35 projects (and they won't), we don't have the internal mentoring resources to guide that many Summer of Code projects.

Colin Charles has already been quite busy replying to the incoming flood of applications, insisting on them being more specific than "I'm in, just give me any task!". Still, some applications are a bit too general for us to be able to pick them. For the others, we will group them into projects that


  1. affect the MySQL Server code base

  2. affect the code base of a MySQL Connector

  3. affect the code base of a MySQL GUI tool

  4. expand or improve the test suites of MySQL Server

  5. introduce new code bases for MySQL AB itself

  6. involve the code bases of entities in MySQL's ecosystem



where the last category is quite an interesting one, where both Sheeri Kritzer and Paul McCullagh have signed on as mentors and have cool projects described on the Forge Summer of Code page.

Expect us to be done next week, in time for the Google Summer of Code deadline for picking projects. In the meantime, we will be quite busy internally pairing up the tasks with the most suitable mentors.

Saturday, 24 March 2007

Building bridges to PostgreSQL

Through Kristian Köhntopp's blog, I ran into a funny posting about "The real difference between MySQL and PostgreSQL" on Andreas Scherbaum's mostly PostgreSQL related blog. I've taken the liberty below post a cropped version of the central picture of that blog entry:



The picture shows Susanne Ebrecht and Lenz Grimmer at the Free and Open Source Software Developers' European Meeting FOSDEM 2007 in Brussels in February. Susanne hosted the PostgreSQL booth at FOSDEM, and Lenz represented MySQL. Drawing a seemingly obvious conclusion from the picture, one of Andreas's blog commenters noted that "MySQL is from Mars, PostgreSQL is from Venus."

Since last month, things have developed. Susanne has gone through a long interviewing process with MySQL AB, where I was one of the MySQLers who had the pleasure of interviewing her. Susanne has now joined MySQL, working for Georg Richter on the Connectors team.

Her joining provoked a number of knee-jerk reactions within MySQL AB. Some (but very few, I am happy to note) recent recruits got run over by their killer instincts and proclaimed some kind of crusade victory. More experienced long-time MySQLers (quite a lot, I am happy to note) instead turned their killer instincts towards these outbursts, explaining the basics of MySQL / PostgreSQL coexistence in the Open Source world.

So let me publish part of my internal message on Susanne's joining. I've censored bits and pieces that would identify whom I was addressing and what that person was saying:

Like you, I'm very happy that Susanne joined us, but her task is not to "kick out Postgres" [..]".

I envision Susanne as someone building bridges between the MySQL community and the PostgreSQL community, rather than as a "secret agent" swapping sides. I understand [... some named prominent PostgreSQL community members ...] don't see her as a black sheep, nor would that be for the benefit of either PostgreSQL or MySQL.

So to make my Community perspective clear: MySQLers shouldn't in any way think of or publicly portray Susanne's joining as "a prominent PostgreSQL person has now changed loyalties".

On the other hand, I can see several good things coming out of Susanne joining us:


  • we can learn to understand Postgres better

  • we can understand Postgres contributors better

  • we can grow our network towards Postgres and its contributors

  • we can publicly underline non-hostility between MySQL and Postgres



Susanne: Welcome to MySQL!

I hope the PostgreSQL community sees similar / reciprocal advantages of building bridges towards us at MySQL.

High Availability: DRBD rcks

On Thursday/Friday this week, I visited Linbit in Vienna. They are the creators of DRBD. Quoting Wikipedia,

DRBD is an acronym for Distributed Replicated Block Device. It is a Linux kernel module, that, working together with some scripts, offer a distributed storage system, frequently used on high availability clusters. DRBD works as a kind of network RAID.

This means DRBD can give high availability to MySQL users. Through configuring DRBD to be used on your system, you can have synchronous replication between two different servers, giving a MySQL database a failover server to redirect to instantaneously, should the main server running MySQL fail.

For those interested in more detail on how to combine DRBD and MySQL, let me mention that Kristian Köhntopp of MySQL has written a great blog article on "Quick tour of DRBD".

I was impressed when listening to DRBD's main author DI Philipp Reisner describing the technical workings and business opportunities of DRBD. In many respects, he reminds me of our very own Monty years ago.

I also learnt plenty of things from Florian Haas, Senior Software Engineer with Linbit. Among other things, he taught me that r is a vowel (in many of Austria's neighbouring countries), meaning that you can pronounce DRBD without spelling out the letters. Sounds like "Good day!" in Slovenian.

On a more serious note, I think the prospects for DRBD look fascinating. Or in other words, remembering my recent insight on vowels: DRBD rcks!

Why the Architecture of Participation is compatible with commercial interests

In his well-written blog entry Open Development: Diversity matters, Gianugo Rabellino quickly replied to my blog entry from yesterday on Defining "Participatory Open Source". He sees plenty of common ground in our reasoning, but defends the existence of requirements for neutrality in the definition. I agree with nearly all of his reasoning for why neutrality is important for the development of a community of contributors, but I draw partially different conclusions.

The main reason why my conclusions are different from Gianugo's is the starting point for my reasoning: There is no inherent conflict of interest between participatory open source and pursuing for-profit business goals, so we must not create artificial obstacles for business wanting to develop Open Source software building upon an Architecture of Participation. Now, I can hear Gianugo objecting "But my entire blog entry and reasoning about why we need an impartial body and non-discrimination was about documenting the very existence of that inherent confict of interest". Well, read on, and let me start by underlining where I agree with Gianugo.

First, I agree with Gianugo in that he has identified the right passage in my previous blog where our reasoning differs:
The interesting middle ground is whether it’s OK to say “no” or “maybe later” to technically sound contributions that don’t fit with the business interest of the owner of the main code base. I would argue that such “discrimination” is OK, as the “Participatory Open Source” requirements would otherwise impose so severe purity limitations, as not to be interesting for companies with a commercial agenda, like MySQL AB.


He argues that this documents that MySQL doesn't have a neutral point of view when accepting code contributions, and I agree this is coherent with the rest of his reasoning.

Second, I agree with Gianugo in that turning down a contribution for no good reason is an excellent way of demotivating a contributor. If I know my contribution is technically highly merited, usable for many, and thus expect it to be included as part of the body code, I will get very upset if it is rejected for apparently no good reason. I won't contribute again, and I will share my anger not just with my friends, but with the general public. Others will likely also be discouraged from contributing. This undermines the Architecture of Participation, and any good community vibrations will asymptotically approach zero degrees Kelvin.

While this seems as total agreement between Gianugo and Kaj, how can we then arrive at different conclusions?

Let me describe a scenario where I think it should be legitimate for MySQL to reject a technically valid contribution on commercial grounds, and why I don't see it as an inhibitor for creating a vibrant community. I can even make the case fairly generic.

Basic assumption: MySQL has created a for-money-only, commercial offering, which constitutes a compelling reason for some customers to buy something from MySQL AB. This offering is of interest to MySQL users and differentiates our free offering from our business offering. It is the "something more" you get for entering a business relationship with MySQL, where you spend money to save time, instead of spending time to save money. The fact that our revenues are greater than zero is proof that there must be some offering from MySQL that our customers consider meaningful on top of what we offer at no cost to our community. Not all of these revenues are services billed by the hour of MySQL employee time spent. Some services are even based upon the delivery of software, such as the MySQL Network Monitoring and Advisory Service. Let's use the term "the little difference" for what MySQL offers the paying customers beyond what we offer the community of users who use MySQL at no cost.

Now, should you have a contribution that renders the little difference useless, don't expect MySQL to be happy about adopting it.

If Gianugo now says, "See, Kaj, that's my point. If there is not a neutral body governing contributions, then progress is hampered by self-interest. So there cannot be true Open Development, and there won't be a vibrant community.", then I disagree at least with the second part of his reasoning, and my ambition is for MySQL AB through its deeds to prove beyond doubt that one can create an Architecture of Participation around Open Source software, while pursuing for-profit interest, even preparing for an IPO. I MySQL can fulfil the ten requirements I listed yesterday

  1. software released under an Open Source license;

  2. well-defined public guidelines for how to participate in the development process;

  3. a publicly available detailed list of bugs;

  4. a public roadmap with detailed descriptions of planned tasks;

  5. code reviews accessible for the public;

  6. public code acceptance criteria;

  7. active public instant messaging between the developers;

  8. well-maintained public system documentation;

  9. a publicly documented mentoring system for grooming new developers;

  10. a worldwide geographic spread of the developers;


then I think we will have created a vibrant community of contributing developers.

So what happened with the upset contributor mentioned earlier, whose well-merited contribution was turned down because it undermines our business offering, and who told all of his friends about his negative experience, and who blogged for everyone to see about why he won't ever contribute again, and loudly proclaims nobody else should, either? Well, I think that won't happen, and I've actually already communicated why, in the little sentence "Now, should you have a contribution that renders the little difference useless, don't expect MySQL to be happy about adopting it."

While most of the interests of the community of MySQL users are very well aligned with those of MySQL AB the company, there is a small subset which is not. Expect us to happily adopt nearly any technically merited contributions that make MySQL more powerful, easy and efficient to use. Don't expect us to adopt contributions that eliminate or fully undermine the compelling reason to buy our commercial offering.

The virtual Gianugo sitting opposite me now objects: "So why would I as a contributor know, or even be interested in, what you consider to be your business interest, and what will undermine it?". I hear him, I understand his question, I think it is a valid one, and I even have an answer. You don't know our business interest. You would merely need to accept that there is a legitimate interest for MySQL to earn money on something, and the rest is taken care of by the contribution process itself. That contribution process has feedback loops between contributors and MySQL to give answers to contributors to the question "Is MySQL interested in adopting my contribution?", before so much time is spent by the contributor that the answer "I'm sorry, no" will render him or her furious. Remember, the virtual Gianugo sitting opposite me accepts that there are technical reasons why we might see us forced to answer "I'm sorry, we cannot enter your contribution into the code base at this point in time." -- the code might not conform to our coding standards, it might be in conflict with another feature we're just inserting, it might make MySQL too slow, it might not work with Replication, etc. etc. And if I, as a contributor, have spent huge amounts of time on a contribution, expecting it to be accepted into the codebase, I will indeed be furious, no matter what the reason for non-acceptance is. If I have an idea for a contribution that is deemed unacceptable by MySQL AB for commercial reasons, but it's turned down by MySQL while I haven't still spent loads of time on it in the belief that MySQL will adopt it, I will be disappointed, but I'm not going to lead a rebellion.

So this is why the yet-to-be-published acceptance criteria for code patches (item 6 in my ten requirements) will also include commercial aspects. This is why the "well-defined public guidelines for how to participate in the development process" (item 2 in my ten requirements) will state that there must be frequent enough feedback loops between the contributor and the non-neutral body managing MySQL, i.e. MySQL AB, to ensure that a contribution being developed is on track towards acecptance in the code, on both technical and business grounds.

My firm belief is thus that the Architecture of Participation is compatible with commercial interests. Non-neutral owners of Open Source code bases who strive for profit can, legitimately and without a bad conscience, exclude certain contributions on commercial grounds. And they can still establish vibrant contributing communities, by not setting false expectations. I think there is a sufficient number of developers who are willing to contribute code to such projects, because of the many common interests, including the expansion of the world of Free Software.

Friday, 23 March 2007

Defining "Participatory Open Source"

The Open Source thinker and BSD advocate Gianugo Rabellino just made me aware of a year-old blog entry by Susan Wu in Planet Apache on "Should OSI redefine the label Open Source?". As I read the entry, it proposes a term "Open Development" and lists these "first stab" requirements:


  1. an Open Source license, of course;

  2. a non-discriminatory access to the developer’s community;

  3. a well-defined and stated process for people to get involved;

  4. a neutral and self-elected governing body;

  5. (more difficult, could mean having a preferential lane) a neutral party such as a foundation owning the code.



I think Gianugo and Susan are pursuing a worthy cause in defining a software development model that doesn't just release software based on an Open Source license, but also builds upon an architecture of participation. However, for reasons I don't understand, they have included requirements that practically exclude companies like MySQL AB from ever fulfilling them. I am referring to requirements 4 and 5, and partly also 2. I argue those requirements are unnecessary: Open Development is possible even in commercial, for-profit companies.

In order not to steal or redefine Gianugo's term, let me thus propose another term "Participatory Open Source", with the following "first stab" requirements:

  1. software released under an Open Source license;

  2. well-defined public guidelines for how to participate in the development process;

  3. a publicly available detailed list of bugs;

  4. a public roadmap with detailed descriptions of planned tasks;

  5. code reviews accessible for the public;

  6. public code acceptance criteria;

  7. active public instant messaging between the developers;

  8. well-maintained public system documentation;

  9. a publicly documented mentoring system for grooming new developers;

  10. a worldwide geographic spread of the developers;



In some respects, this is similar to Gianugo's requirements. My 1 = his 1. My 2 = his 3. With his 2 and 3, he may also refer to some of my items 3 to 10, that are on a more detailed and practical level, attempting at describing the baseline components of an Architecture of Participation in software development.

In other respects, there are clear distinctions between Gianugo's set of five, and my set of ten. Most of all, my set of ten would work for a company with a commercial agenda (like MySQL AB), whereas Gianugo's items 4 and 5 would not. I think it is absolutely possible to combine a commercial agenda with an Architecture of Participation when developing software, and not just in producing or distributing content (like, say, Google or Amazon). That is why I have excluded any requirements like Gianugo's 4 and 5 from my list.

On a specific, detailed level, I also see a need for clarification in Gianugo's item 2 "a non-discriminatory access to the developer’s community".


  • First, what is "non-discriminatory"? If non-discriminatory means "code contributions shall be accepted regardless of the gender, race, country of origin, or age of the contributor", it's probably OK (whereby certain countries of origin might incur exposure to legal or other risk). If non-discriminatory means "even technically inferior contributions must be accepted", it's obviously not OK. The interesting middle ground is whether it's OK to say "no" or "maybe later" to technically sound contributions that don't fit with the business interest of the owner of the main code base. I would argue that such "discrimination" is OK, as the "Participatory Open Source" requirements would otherwise impose so severe purity limitations, as not to be interesting for companies with a commercial agenda, like MySQL AB.

  • Second, what does "access" mean? The right to participate? Sure. The same rights as the central governing body? No way.

  • Finally, what is "the developer's community"? OK, this may only need a phrasing clarification, as I suspect Gianugo does refer to the community of people developing the software itself (in our case, those who code MySQL using C or C++), as opposed to the community of people developing code using the software (in our case, those who use MySQL when coding in PHP, Perl, Java, Ruby, Delphi, VB, Python, etc. etc.).



At any rate, I fully agree with Gianugo that there is more to Open Source than an Open Source software license: The active participation of the community in developing the software.

Oh, and finally: I don't think MySQL AB is yet fulfilling all ten points. But if I can have my way, we will when the year 2007 is over.

Thursday, 22 March 2007

MySQL as a Research Project Contributor

From time to time, MySQL employees are approached with proposals from universities and other research institutions about MySQL participating in research programmes financed by governmental initiatives in the US and in the EU. This is much appreciated. Often, there is a mutual interest, where


  • academics need an industrial partner to anchor the results of the research into a real-world application and even to fulfil the requirements of getting a grant, and

  • the MySQL user community needs new features in MySQL Server, or in the Connectors enabling the use of MySQL from various development environments.



Left: MySQL University refers to the highest form of education when it comes to learning how to develop MySQL Server (as opposed to using it). It's opening up for the MySQL community to attend, at no cost.

So what do I suggest you to do, if you're drafting a Framework Programme proposal (these are collaborative research projects funded by the European Commission) or equivalent other type of application involving governmental funding, and you think MySQL (whether AB, Inc., GmbH or KK) would be a possible industry partner?


  • First, think about how your research can end up as a benefit for the MySQL user community. Who needs it? To fulfil what requirement? Why is your project going to end up in deliverables that are usable by the MySQL ecosystem?

  • Second, think about what MySQL can do to help. We have limited resources for developing MySQL Server further, and we deploy them carefully. However, we absolutely want to help, if your research proposal provides good answers to the primary questions above. We are more likely to be able to help, the more concrete the contribution of the research partners is.

  • Third, email me or anyone else in the MySQL Community Team, at firstname@mysql.com. Or if you happen to know someone at MySQL, or you know of a MySQL employee close to your geographic location, approach that person -- who in turn will approach the Community Team at MySQL. We will then act as your contact persons and primary interface towards our development organisation, who in turn are the domain experts in Data Warehousing, High Availability, Geographical Data, Clustering, Replication, or whatever the specific domain of your research is about.



While none of us in the Community Team has yet acquired a PhD, we'll connect you with those MySQLers who have -- once we've determined that your research proposal has a common goal with what MySQL is trying to accomplish.

Wednesday, 21 March 2007

Visiting SNAP Innovation, developers of PBXT Storage Engine

I'm just back from SNAP Innovation, the developers of the Primebase XT (PBXT) storage engine, here in Hamburg / Altona. What a pleasure to visit a company that, despite its closed-source roots, has understood and embraced what Open Source is all about and how it can open up business possibilities!

Left: Paul McCullagh

PBXT is a pluggable storage engine that fits in with MySQL 5.1. It is fully transactional, and supports MVCC. That makes it a good fit for many purposes, but the area where it is an especially good fit is for the industry that SNAP has been serving for the past decades: the publishing industry. No matter whether print or web, they have fairly similar needs, where BLOBs are a common denominator.

While MyISAM has supported BLOBs since the 1990s, and while the inventor of the concept of BLOB is Jim Starkey developing the Falcon engine, I can see PBXT satisfying needs beyond the mere moving the storage of pictures, films, MP3 files and other such data off the file system, into the database. Although PBXT isn't there yet, Paul McCullagh (the PBXT author and SNAP CTO), Ulrich Zimmer (their CEO) and Volker Oboda (their director of Mktg & Sales) presented an interesting vision of PBXT as the engine for scalable BLOB streaming. While most access to the PBXT based data will be through MySQL, SNAP seeks to develop a dedicated API (over a separate protocol) accessing the streaming aspects of the BLOBs more efficiently than can be made through a SQL constrained interface.

I'm happy to note that Paul McCullagh will present at the MySQL Conference & Expo 23-26 April 2007, with a session PrimeBase XT: Design and Implementation of a Transactional Storage Engine (Wednesday 10:45-11:45 in Ballroom F), and on the panel in Lightning Rounds: State of the Partner Engines (Thursday 10:45-11:45, Ballroom H). I'm sure Paul will take advantage of the opportunity to integrate with not just MySQLers but also top PHP & Ruby developers, since I think there are some joint interests between PBXT and several development environments.

We talked about various aspects of better integrating PBXT into MySQL and our community activities. Let's see if Paul manages to put up a task for MySQL's Summer of Code page, for benchmarking of the PBXT engine.

Read more about PBXT in Robin Schumacher's article "A look at the PBXT Storage Engine". If you want to follow the ongoing development of PBXT, consider reading Paul's blog (which is also aggregated on Planet MySQL) and take PBXT for a test drive yourself by downloading it from http://www.primebase.com/xt/.

It seems to me that SNAP intends to build up a sizable community of users of the PBXT storage engine under MySQL, and that they're in it for the long run, aiming for their scalable BLOB streaming backend.

MySQL Global Sports Day

Our friendly Human Resources department started a new initiative this week: We have a Global Sports Day for all our >360 employees in >29 countries. That means that us MySQLers are to get off the keyboard, move our bodies, and document our activities for the rest of the company to see. We are reimbursed for expenses, and the most creative activity / picture / story will get an additional reward.

Lenz and I happened to have planned a Community meeting for today already quite a while ago, so we decided to combine this with the Sports Day. Our local version in Hamburg took us 5,43 km through Niendorfer Gehege in 0:31:35. For myself, this means that I have run a total of 161 km so far this year, for about 15 hours, at an average speed of 10,3 km (I'm sure you're interested). For Lenz, this was the first time he moved himself this year out of "free" will on a bike.




Lenz getting ready on bike ...... and Kaj on foot.



Kaj running ...


... and Lenz biking.


Getting dark in the Niendorfer Gehege.


Kaj and Lenz getting closer to the goal.

The reason I know the exact distance of 5,43 km is that I bought a Garmin Forerunner 305 earlier this year. I fear this gadget won't be within the budget of the Sports Day reimbursement, but it's great fun for a running geek. Hey, if it motivates me to run more, it's worth it! Perhaps I should read up on the notes from MySQL Camp last November about how the Motion Based guys (a Garmin subsidiary) are using MySQL for spatial data.

Thursday, 15 March 2007

MySQL Summer of Code: Apply now, window closes 24 March 2007!

As I wrote a few days ago in my blog, MySQL is participating in the Google Summer of Code 2007.

The application process for students has now been opened! You have just one good week of time to apply, as the window closes 24 March 2007.



To participate in MySQL Summer of Code,

  1. Read my blog announcement on http://www.planetmysql.org/kaj/?p=89

  2. Read Google's Guide to the GSoC Web App for Student Applicants at http://groups.google.com/group/ google-summer-of-code-announce/ web/guide-to-the-gsoc-web-app-for-student-applicants
  3. Go to the http://forge.mysql.com/wiki/SummerOfCode page to pick one or several MySQL related projects that interest you

  4. Load the http://forge.mysql.com/wiki/SummerOfCode/ApplicationTemplate page for the MySQL related application template, to use as a reference for when you fill in the application itself in the next step

  5. Go to the Google Summer of Code Student Signup page at http://code.google.com/soc/student_step1.html and start the signup process itself. Be prepared that it can take a while to fill in.



When filling in the GSoC application, remember that we expect you to follow our Application Template mentioned above. I.e.

  • Your application needs to be concise.

  • Your application needs to follow our format.

  • First you give us your personal details,

  • then what you plan on doing,

  • then all your actual experience (broken down, for easy reading),

  • then what exactly you intend on delivering (the most important part), and

  • finally, a simple Yes, I'm willing to sign the MySQL CLA at http://forge.mysql.com/wiki/MySQL_Contributor_License_Agreement



If at any point you need assistance, please don't hesitate to contact MySQL's Summer of Code Project Administrator Colin Charles (colin@mysql.com).

Wednesday, 14 March 2007

MySQL Conference & Expo Early Registration closes today

The early registration of the MySQL Conference & Expo 23-26 April 2007 closes today, 14 March 2007. So if you want to cut your attendance costs by 200 US dollars, register now!



Quoting the About The MySQL Conference & Expo page:

The MySQL Conference & Expo will feature:

  • Over 100 sessions for a wide range of skill levels

  • Tutorials to help develop technical skills and learn new tricks

  • More than 2000 MySQL developers, gurus, experts, and users under one roof

  • Dozens of new tools and technologies from third party vendors in the Expo Hall

  • Birds of a Feather sessions, evening events, and networking opportunities.



That listing obviously misses the most important point:

  • The Clash of the DB Egos, a provoked fight between some of the most opinionated two-initials DB gurus the world has seen



I'd love to see you there!

Monday, 12 March 2007

MySQL 5.0.37 Community Server, Contributions and Binaries

Finally!

We have a release of MySQL Community Server that contains community-provided contributions and it is ready for your download in binary form.

Thanks to Jeremy Cole for providing us with the SHOW PROFILE patch. Quoting our Reference Manual at http://dev.mysql.com/doc/refman/5.0/en/show-profiles.html:


SHOW PROFILES

SHOW PROFILE [type [, type] ... ]
[FOR QUERY n]
[LIMIT n [OFFSET n]]

type:
ALL | BLOCK IO | CONTEXT SWITCHES | CPU | IPC | MEMORY | PAGE FAULTS | SOURCE | SWAPS


The SHOW PROFILES and SHOW PROFILE statements display profiling information that indicates resource usage for statements executed during the course of the current session.

Profiling is controlled by the profiling session variable, which has a default value of 0 (OFF). Profiling is enabled by setting profiling to 1 or ON:

mysql> SET profiling = 1;
SHOW PROFILES displays a list of the most recent statements sent to the master. The size of the list is controlled by the profiling_history_size session variable, which has a default value of 15. The maximum value is 100. Setting the value to 0 has the practical effect of disabling profiling.

All statements are profiled except SHOW PROFILES and SHOW PROFILE, so you will find neither of those statements in the profile list. Malformed statements are profiled. For example, SHOW PROFILING is an illegal statement, and a syntax error occurs if you try to execute it, but it will show up in the profiling list.

SHOW PROFILE displays detailed information about a single statement. Without the FOR QUERY n clause, the output pertains to the most recently executed statement. If FOR QUERY n is included, SHOW PROFILE displays information for statement n. The values of n correspond to the Query_ID values displayed by SHOW PROFILES.

The LIMIT n clause may be given to limit the output to n rows. If LIMIT is given, OFFSET n may be added to begin the output n rows into the full set of rows.

Some sample output:

mysql> SHOW PROFILES;
+----------+----------+--------------------------+
| Query_ID | Duration | Query |
+----------+----------+--------------------------+
| 0 | 0.000088 | SET PROFILING = 1 |
| 1 | 0.000136 | DROP TABLE IF EXISTS t1 |
| 2 | 0.011947 | CREATE TABLE t1 (id INT) |
+----------+----------+--------------------------+
3 rows in set (0.00 sec)

mysql> SHOW PROFILE;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| checking permissions | 0.000040 |
| creating table | 0.000056 |
| After create | 0.011363 |
| query end | 0.000375 |
| freeing items | 0.000089 |
| logging slow query | 0.000019 |
| cleaning up | 0.000005 |
+----------------------+----------+
7 rows in set (0.00 sec)



This information is also visible in the corresponding INFORMATION_SCHEMA.PROFILING table.

Unfortunately the SHOW PROFILE implementation as of 5.0.37 still has a small gotcha, that we will resolve for the next release: the MySQL command line client hangs if you issue the SHOW PROFILE command without having enabled the @@profiling session variable before. Hence, when you evaluate this new feature, make sure to enable it by issuing "SET @@profiling = 1;" before.

In summary, the SHOW PROFILE patch provides a way for the MySQL Community Server users to get a more detailed understanding of where the response time is formed. Jeremy also contributed a patch to display the Uptime_since_flush_status status variable, which indicates the number of seconds since the most recent FLUSH STATUS statement. Thanks, Jeremy!

Looking beyond the community enhancements, MySQL Community Server 5.0.37 is our second full (source and binary) release of the MySQL Community Server since we made the split between the Community and Enterprise Version in October 2006. It includes all bug fixes applied to up to and including the MySQL 5.0.36 Enterprise Server. The prior MySQL Community Server 5.0.33 release early January 2007 was a source-only release.

The 5.0.37 release also resolves a crashing bug that could be exploited as a potential local Denial of Service attack. Thank you, Stefan Streichsbier from SEC Consult, who also informed us about this bug via our security@mysql.com mail address - he will send out a separate advisory about his findings.

We welcome and appreciate your feedback, bug reports, bug fixes, patches etc. as described on MySQL Forge on http://forge.mysql.com/wiki/Contributing.

Community Engineering Team Established

The MySQL Community Team has been strengthened, getting an Engineering arm able to process code patches.




Giuseppe MaxiaChad Miller


In an effort to align MySQL company resources to support an Architecture of Participation, we have reallocated Chad Miller and Giuseppe Maxia from the Engineering team to form the Community Engineering team. Chad is allocated towards enabling us to quickly review code patches (bug fixes, new features) and Giuseppe towards our Quality Contribution Program.

Looking at this from a Community perspective, this forms the following step in the process of opening up our Development Model and supporting the Architeture of Participation. Looking at it from an internal company perspective, this forms the following step in the specialisation of resources -- a development which started more than five years ago when we dedicated some of our engineers mainly to Support, while others remained in Development.

No matter from which perspective you look at it, the establishment of the Community Engineering Team is intended this to increase our level of responsiveness to community contributions. And that is an area where we still have room for improvement. Our bugs system contains some skeletons in the form of pending community-provided patches, such as the performance enhancements of Gunnar von Boehn. The Engineering department has been busy fixing bugs according to certain priorities (which aren't wrong per se), and the Community department has lacked the technical expertise to be able to safely apply patches. Now, we have an organisational unit which has as its main priority to work for and with community contributors, and which has the technical skill set required. This unit, Community Engineering, only depends on the rest of Engineering for reviews.

I've already mentioned it in my Summer of Code blog entry, but it bears repeating: We've seen some early wins based on this organisational change. One of our top Quality Contributors, Martin Friebe, submitted a bug report including a test case and a bug fix code patch. Through the allocation of Giuseppe and Chad to Community Engineering, the patch was verified and applied to the tree within two hours of the bug report. I hope we are able to continue down this road!

Welcome, Giuseppe! Welcome, Chad! And thanks to our VP Engineering, Jeffrey Pugh, for letting me have them!

Global Warming & Google Summer of Code

What happened with the MySQL Winter of Code? It was hit by Global Warming. Specifically, the Community Team got heated up working on the MySQL Conference & Expo 23-26 April 2007, left with little bandwidth to pursue the ideas and action items for Winter of Code.

When winter warms up, it slowly turns into summer. But the coding work remains. Colin Charles was smart enough to combine these two facts with Sheeri Kritzer's blog entry and hints about the Australian Winter, and took action related to the Google Summer of Code. Colin spoke to Chris DiBona and Leslie Hawthorn of Google and has now signed us up for Google Summer of Code.

In short: We are transitioning the Winter of Code to the Google Summer of Code, and we today sent in our application to Google.

While I am not happy about our not having been able to devote the necessary resources to our WoC, I am happy that the Quality Contribution Program has been launched and had some early successes -- like when Martin Friebe submitted a bug report including a test case and a bug fix code patch, which enabled usto verify the bug and apply the fix within two hours of the bug report. Thanks Martin Friebe, Giuseppe Maxia and Chad Miller!

In an attack of misguided pride, we at MySQL AB could choose to feel bad about not having found or fixed that bug ourselves. I don't. Bugs are a fact of life, and it's our task to architect our model so that they're fixed fast. We hate bugs! And we love those who help us kill them!

The Quality Contribution Program has oiled our internal processes for being able to receive, process, review and accept external patches -- something which we traditionally haven't been good at.

In fact, we haven't been creating as much of an Architecture of Participation at MySQL, as we should expect. What's an Architecture of Participation? Let me quote the two first sentences from Wikipedia:


The phrase architecture of participation describes the nature of systems that are designed for user contribution, such as open source and Wikipedia. It was coined by Tim O'Reilly, who described it at length in a 2003 speech and later in several of his online writings.

The phrase has come to define one of the key elements of what's been called Web 2.0, which describes the collection of companies, technologies and projects that are designed around the culture and economics of openness.


Now starts the next step in opening up the MySQL Development Model. The two things that SoC require of us are the same as what an Architecture of Participation requires of any organisation:

(i) identifying mentors
(ii) identifying features to implement

Identifying mentors: Experienced developers need to "Adopt A Programmer" and guide these contributors on the MySQL source code, and how to implement the features they (or we) want them to develop. This takes several hours a week, but if well done, it's as rewarding for the mentor as it is for the students being guided.

While folks running MySQL University are ideal mentor candidates, I would argue that students at MySQL University also qualify. We absolutely want more than the topmost ten hotshot developers at MySQL as mentors.

Identifying features: In the tentative Winter of Code plans, we proposed large, ambitious tasks


  • Improvements to (pure drivers for) Perl, Apache APR, Python, Ruby or new connectors for emerging environments

  • New Storage Engines: The Mail Inbox Format Storage Engine, the File System Storage Engine for Windows, for Mac, for Linux, or the Digital Picture (JPG / EXIF) Storage Engine



However, for SoC, we have tasks of all ambition levels -- and have not limited ourselves to entirely new features. Jeffrey Pugh, our VP Engineering, and his Dev Leads have provided Colin with a digested list, that Colin is managing at http://forge.mysql.com/wiki/SummerOfCode and that currently has the following item headers


  • Test Suite Development (Stewart Smith)
    mysql-test-run.pl and mysqltest

  • Test case development (Giuseppe Maxia)
    Code coverage improvement
    System Tests - Load Tests and Long-Running Tests
    Test creation tools

  • Benchmarking the MySQL Server

  • Instance Manager fixes

  • Integrate MySQL Cluster with Instance Manager

  • INFORMATION_SCHEMA tables for MySQL Cluster status

  • MySQL Based Atom Store

  • Simple P4 bugs/features involving options/flags on bugs.mysql.com

  • Simple features from Worklog, our detailed todo list



At last query, the MySQL bugs system had over 1,499 bugs filed in a Priority 4 state, which means they're Feature Requests (Severity 4). While all of these are "up for grabs", we don't expect our Summer of Code students to pick their favourites from such a long list. Soon, we'll rank the top ones based on our judgement of how interesting the features are to use, and how easy they are to develop.

So, if you're interested in participating in our Summer of Code, what should you do?


  1. Look for interesting projects in our project list on MySQL Forge.

  2. Fill in an application according to our application template, also on MySQL Forge. Do this on the Google Summer of Code web app that will be live from 14 to 24 March 2007.



And if you have questions, feel free to contact Colin Charles and/or myself over email, firstname@mysql.com.

Friday, 2 March 2007

The Clash of the DB Egos

Would you like to experience a battle between E. F. Codd, C. J. Date, U. M. Widenius, J. A. Starkey and U. M. Ronström? Some of the biggest egos the database world has seen!

Edgar 'Ted' Codd

Well, I cannot promise you all of the above, especially when it comes to Codd, the father of relational databases (1923-2003). Codd's colleague Date might also be a no-show, but I think I can promise you Widenius (MySQL, MyISAM), Starkey (Datatrieve, Interbase, BLOBs, MVCC, Falcon) and Ronström (NDB Cluster). During the Wednesday keynote at the MySQL Conference & Expo 23-26 April 2007 in Santa Clara, I will be provoking a fight about what constitutes a database, what's the right way of doing locking, whether it ever was smart to store BLOBs in a database to begin with, and who ever needs versioning.

Register Now & Save!

Given how I know said egos, I don't expect it to be hard to throw in a couple of sentences that will fire off a slight disagreement between the gentlemen. And during the course of the discussion, I plan to bring in a few more egos, from Solid and Nitro.

Let the battle begin!

MySQL Dev Mtg Heidelberg 18-24 Sep 2007

In my blog entry about How to arrange a physical meeting in a virtual organisation, I indicated that MySQL AB would arrange its next Developer meeting early in May 2007. For various reasons, including avoiding a crash with the MySQL Conference & Expo 23-26 April 2007 in Santa Clara, we postponed it to September. Our finalists were Sofia, Heidelberg, Munich and Vienna, where Germany's Next Top Dev Mtg Location was choosen to be Heidelberg, one of Germany's most charming cities ideally located at about one hour's shuttle distance from Frankfurt Airport.

Coat of arms of Heidelberg The castle (Schloss) above the town

As part of our 2007 overall theme to open up our Development Model, we expect to open up attendance for some external community members. If interested, please email both Lenz Grimmer (Community Relations Mgr, Europe) and myself at firstname@mysql.com.

Coincidentally (honestly!), the Developers Meeting ends on the Tuesday right after the world's biggest party Oktoberfest has started. So whoever joins us should consider flying into and out of Munich instead of Frankfurt, and spend 3 hours each way on train. And book early, since another 5 999 999 visitors are expected in addition to yourself.

Thursday, 1 March 2007

MySQL University, MEM_ROOT and Pillows

I'm listening to the first MySQL University session. It's the highest form of education available at MySQL AB, hence the name. Not about how to use MySQL, but how to write code patches for MySQL.



Right now, Sergei Golubchik is sitting in Cologne, Germany, speaking behind Skype Conferencing and two pillows (sound quality!) about memory management and mysys libraries and algorithms. Monty and others are assisting Sergei over our internal IRC, where there are 31 developers listening in.


Our presenter, Sergei Golubchik, in a non-virtual setting

Here's an IRC transcript:


monty|uni ie, only destructior is called, memory is not freed until free_root()
rafal|school monty|uni: if I initialize String from other String, or char table[] or "string constant", is memory allocated/deallocated correctly?
iggy|uni serg, While talking about alloc/dealloc for Strings, could you comment on the usefullness of SAFEMALLOC and how to take advantage of it?
monty|uni rafal|school: should be
rafal|school monty|uni: ok (thanks for all answers)
monty|uni Note that String can be both a pointer (like char*) and a C++ like string that holds the string itself
monty|uni If you use it as a pointer, you don't get it freed. If you use it as a string (or convert it from pointer to string with String.copy(), you need to free it
monty|uni Using delete works on both cases
monty|uni so to be safe, you should just call delete on all String objects.
rafal|school monty|uni: how String can be used as a pointer?
mats|phone monty|uni, So that means that you have to allocate memory for a C string if you are passing it into the String constructor?
monty|uni Default is pointer
monty|uni mats|phone: or you use string.copy()



It's a challenge. I'm not talking about understanding the routines, like


  • init_alloc_root() - Initalize a MEMROOT object

  • alloc_root() - Allocate an object

  • multi_alloc_root() - Allocate many objects at once

  • free_root() - Free everything



nor about understanding Serg's accent. Everyone at MySQL has an accent. Our internal language is Bad English, and I feel competent to conform.

The challenge is the low bandwidth. I'm also sitting in Germany, and sadly, it sounds as if Serg were speaking through two pillows.

So why am I writing about this? Because we are planning to open up MySQL University. We are thinking of moving the IRC chat to freenode, and the Wiki pages to the MySQL University Library on MySQL Forge Wiki. The sound already is public, on Skype Conferencing.

With the open MySQL University, those who have an interest in learning about MySQL's internal code structure can listen in, or follow the MP3 recording later on, or just browse the MySQL University Library. Stay tuned -- we hope to be able to open up these weekly sessions later on in March.