Drupal Cleveland Blog

Companies and Contributing to Drupal

Jeff Schuler — Thu, 23 Jul 2009 18:15:51 +0000

The Lullabot Podcast is a great Drupal resource -- from a rockstar company comprised of some of the very best of the Drupal community.

I was listening to an old episode, (because of a Twitter from DavidWheelerPhd) and came across a quote that resonated with me:

The other thing that I've heard a lot of businesses say is that they can't find Drupal people fast enough.

What Drupal developer, or contributor in general, doesn't want to work for a company who gives stuff back to the community?

Because that's the ultimate thing, right? You want your work that you do to be done for tons and tons of people -- because it's good for your feeling of self worth, it's good for "the world..."

If you have a choice between signing up with a company who does a lot of work and gives it back to the Drupal community, and helps everybody, or going with a company that makes you sign a bunch of NDAs and locks everything up in perpetuity, and the minute you leave that company all your code is gone...

It's a no-brainer choice.

-- Angie Byron (webchick), Lullabot Drupal Podcast No. 54: Contributing to Drupal (at about 33:44) [my transcription & edits]

The health of the project and the world at large are super important factors to the workings and decisions of the Drupal community. This is the Long View, this is sustainability.

Drupal Presentation for WebSIG

Jeff Schuler — Sun, 19 Apr 2009 02:46:59 +0000

Attached are PDF and OpenOffice Impress versions of the Drupal presentation I prepared for the WebSIG Open Source Content Management Systems: Joomla! & Drupal meeting today.

The slides comprise a brief overview on Drupal, plus-and-minus certain particulars. Any comments and feedback are welcome and appreciated, as I may refine and expand this for future use.

I'm releasing the presentation under the CreativeCommons Attribution-Share Alike license -- feel free to reuse and adapt, but please give credit and redistribute likewise.

Thanks to everyone who attended for your friendly and active participation!

Attachment	Size
WebSIG.pdf	84.35 KB
WebSIG.odp	65.45 KB

A Glimpse into Drupal's Open Source Process

Jeff Schuler — Thu, 12 Mar 2009 00:14:30 +0000

Drupal is free and open-source software. Anybody can download, use, modify, repackage, and redistribute it (event at a price,) freely. Open source is not only key to its distribution but also its development, culture, and very nature.

Drupal is modular to its core, meaning it's made up of relatively small pieces, loosely joined. Adding a new feature to a site is often accomplished by installing or creating a module to do so. The collapsible sidebar menus on this site are driven by a module called DHTML Menu, there's a Google Analytics module for tracking traffic, I just installed SaveGuard to warn before allowing to browse away from a page when editing a post, and the list goes on... Blog is a core module, and Aggregator, and Forum -- these are the modules that comprise Drupal itself, and come standard -- out-of-the-box, (though thankfully there's no box.)

A big part of Drupal's major releases, (this site is running version 6, a number of other sites I'm administering are still on 4 and 5,... and 7 is under development,) is the rolling of contributed modules into to core. For example, Drupal 6 brought the OpenID module into core, and Drupal 7, CCK. These modules were originally created by developers who had an itch to scratch -- maybe for a client, maybe their own interest. CCK was contributed in 2006, and is now the second most-used contributed module. OpenID, begun the same year, is much further down the list in popularity, but decided an important inclusion -- for the evolution of Drupal in the emergent distributed online identity space. The community chose that these modules should become part of the basic Drupal installation, instead of discovered and installed separately.

When a module is contributed, it becomes part of the open source process, and managed on Drupal.org. Its project page, (e.g., http://drupal.org/project/rotor ) aggregates info on what the module does, and how to download and use it, as well as provides a platform for feedback and contribution.

A module's Issue Queue is typically the first place to look to see how the module is faring in the wild. Since a contributed module is owned and co-created by the whole community, issue tracking is central to its evolution. Bugs, typos, feature requests, and problems are all submitted and handled here. If I find an error in a module's workings, I submit an issue about it.

Each module has a maintainer, who may or may not be its original creator. An important distinction exists between developer and maintainer -- the latter of which is largely responsible (as facilitator/manager/leader) of the module's evolution, but not necessarily the one writing all of the code to improve it.

If I submit an issue on a module, someone else may then come along and corroborate what I've seen, another person suggest a solution, and yet another write code to patch the issue. The community then reviews and tests the patch and, if approved, the maintainer might then commit the patch to the module. The change has now been incorporated into the module's development version, waiting for further inspection to be marked as 'stable.'

The process can be even more involved at every step: a number of designations of an issue's state exist, and quite a few issues have been around for years, yet without an agreed-upon or implemented resolution. Drupal's core happens on the same system, in the same manner.

Issues, and comments on them, are on the order of hundreds of thousands. Constant discussion, support, and collaboration happen in IRC (chat) channels #drupal, #drupal-support, #drupal-dev, and many more. The ecosystem of support functions well. Its strongest members lead with the realization that handing out ownership and encouragement inspires investment, responsibility, and great work: a meritocracy and gift-economy. If you use Drupal, are you contributing back? You might not realize how easy it can be. It's not just your responsibility or the good feelings, it's your community and your co-creation.

It's amazing and refreshing that an ad-hoc and decentralized group works so effectively. Co-creating a more secure, fault-tolerant, distributed-risk, adaptable, and robust system: sounds like a sustainable solution to me.

The Path to Richer Text (DrupalCon DC 2009 notes)

Jeff Schuler — Fri, 06 Mar 2009 22:17:58 +0000

The Path to Richer Text: Input Format Improvements in Drupal 7
David Rothstein

Adding input formats to main permissions lists -- this role can use this input format. Not quite in D7 core yet.

Warnings on permissions page for allowing unsafe formats? New patch needed.

Must get rid of allowing setting of site-wide default text format.

Weights on input formats

User's default input format is her lowest-weight (highest on list) allowed format
Plain text becomes site-wide lowest-common denominator text format

Long standing issue with patch (PCNR -- please review!) for this: #11218: Allow default input formats per role, and integrate input format permissions.

Also be easy to reliably (in code) override the default format, and user experience improvements.

Text format widget - adds list menu of available formats below text field (instead of

A particulary WYSIWYG included in core is not the right question to focus on now. Better WYSIWYG support is. Check out sun's WYSIWYG module and g.d.o./wysiwyg.

Can D7 text formats give WYSIWYGs enough info to disable buttons on their own?

Can we get D7 core/contrib modules to include text formats attached to them?

With Fields in Core! (patch posted yesterday: #391924: Use Fields to store variables that require text format support )

Fields have text format support built in, and you can attach a field to any kind of "object"
..Was relatively easy, then to create an API that uses fields to automatically store formatted text

Keynote: Our Online Identity (DrupalCon DC 2009 notes)

Jeff Schuler — Fri, 06 Mar 2009 20:22:57 +0000

Keynote: Our Online Identity
Chris Messina

Social Web, Cloud computing, and its discontents

Our Online Identies, My Online Identity, I am Mine, I am You

Web 2.0: WTF?

"Build applications that harnes network effects -- to get better, the more people use them." (O'Reilly)

Rules

The perpetual beta becomes a process for engaging customers
Share and share-alike, reuse, provide APIs
Ignore distinction between client and server
Open and standards win
Lock-in comes from data accrual, owning a namespace, non-standard formats

World doesn't really operate on version numbers.

Does Open Source Matter?

Not like it used to.
it's more about data...

Technology is becoming humanized. Humans are bcoming technologized.

Before blow-ups and big buy-outs, we (geeks) were outliers using little unknown sites and gadgets -- . Now, those who don't, are becoming the outliers...

We are becoming cyborganic.

Summary

Identify
- online services I use help defines [part of] my identity
- social graph also helps define: you somewhat the aggregate of your friends
- (false) dichotomy between online and offline world is going away
- Online Social Networks are moving away from "handles" to Real Names -- this is significant
Friends
Social Web
Open, Social Stack
Privacy and Property
Cloud Computing

#hashtags: low-cost way to add to a service without changing underlying premise

Activity Streams

enhance way to utilize social web

Twitter usage has been hockey-stick growth, whereas blogging doesn't work as well, now that so many folk are online -- now not enough time to write and write and read and read.

We're addicted to our friends, not the computer. When the computer lets us get access to our frineds, we look like we're addicted to the computer. -- danah boyd

moving away from banal senseless befriending behavior to more pointed attention. Activity streams allow and necessitate this.

How do you get your service into the FriendFeed list? RSS/Atom only way, right now, but these feed types are not enough to model richness that web offers.

Creating an extension to atom that allows for more expressiveness:

Weblog Entry
Note
Photo
Video
Bookmark

Move out of Facebook and FriendFeed, but allow for rich activity streams

Open, Social Stack

In Drupal, Activity module coming along..

The Stack, (human readably) : Identity, Friends, Feed

The Blimp diagram: Identity contains Discovery contains Authorization

Private and Public data: your Streams, Friends, Profile have some parts needing auth, some not

Facebook gives pretty robust control over what you show to who, with assumption that options make users feel safer and more in control

Performative Identity

People, companies, and organizations are discovering that there is value in openness.

Legality of sharing user information? All bets are off.

People want full ownership and control of their information so they can turn off access to it at any time. At the same time, people also want to be able to bring the information others have shared with them—like email addresses, phone numbers, photos and so on—to other services and grant those services access to those people's information. These two positions are at odds with each other.

Mark Zuckerberg, On Facebook, People Own and Control Their Information

Walled gardens are going away.

Licensing and "property"

copyrights, trademarks, patents: protect creators

Ownership, but ability to expose to rest of the world

express where you store your data...

Government

How to bring transparency to gov't?
- current situation is result of lack
- we (geeks) know more about transparency than most others
Open Source Government
- Transparent and modifiable government
OpenID in Government
- Trust and transparency make open source work

Work on stuff that matters!

Work on something that matter more to you than money.
Create more value than you capture.
Take the long view.

Project Management For Fun and Profit

Jeff Schuler — Fri, 06 Mar 2009 19:21:28 +0000

Project Management For Fun and Profit
Crystal Williams

The Talking Stage

There is no shortcut
Clients are the experts on their needs, though they don't always know how to express that, but your'e the expert on getting them there. Show what you know, but listen.
You can't afford to save time on planning or communication.

Process

Your toolbox: light and agile but provide structure
minimize the "what now?", "where is that file?", "when is it due?"
re-examine until you have structure works
there should be an owner for every piece: if you're a PM, try to make it not be you!

Agile vs/ Waterfall

Neither is appropriate for ALL projects or developers, so be careful.
Agile
- has become a sacred cow in OSS community. Can be great when it works, but takes a lot from client
Waterfall: "standard" sequential design & dev process
Holy war...

Good for:

Agile
- User-centric social networking sites, Web apps
Waterfall
- Corporate Sites and other informational sites, even with interactivity

Agile Requirements

Extremely talented and motivated developers who know how to say No
Constant access to design resources
Progressive clients who are willing to invest continuous resources in the project and actively think about the direction

Even with Agile,

Estimating

is hard!
The CivicActions Estimating Worksheet
State assumptions
Estimate each area of work separately, and include adequate time for communication with client and internal, and testing
Never estimate "to" a budget: line items disregard available budget: how long to get the job done?
- discuss a reduced feature set

Scheduling

Mythical man-hour: 8 hour task doesn't take 2 people 4 hours.
order of operations: gaps in dependencies
client approval time

Design

Clients don't care about Drupal. Design for their needs, and not for Drupal defaults.

Tools

scheduling: Merlin, MS Project
WIreframes and sitemaps: omnigraffle, Visio
Shared notes, estimation worksheets, proposal drafts: intranet, wikis, google docs
task and time tracking: basecamp, unfuddle, storm, harvest, LiquidPlanner, more more
Bug tracking: Mantis, Trac, JIRA, Case Tracker Module, many
Code Review: Crucible
Whiteboards (real)
Skype/IM : sparingly, and never for critique
Phone: don't ignore!

Clients

Great project do not happen in spite of bad clients
seek and cultivate great clients
Great work to show off takes clients who 'get it' -- and are stable enough to support their end of the deal.

Warning Signs

"I don't care, just get it done"
"I don't need project management"
"I need it this week"
"I could get a freelancer [in India] for $__"
"My neighbor, [who's an expert in this,] says..."

Good Clients know they're investing. Key questions

Immediate goal or cricital biz issue: reason for the project? can they articulate?
Budget: What's budget for the project or the estimated cost of solution? Ample funds for their scope? Ambition is good, but make sure they're stable.
Timeline: When does the project need to go live? External influences?
Key metrics of success? Who decides? They should be important to all.

Basic Client Management

Focus on goal
Have central contact and ensure they're vetting all through necessary stakeholders
You are the General for the web aspect of their brand: act it.
Demonstrate expertise
Know their market, etc
Provide real value

Urgency is a factor

Don't over-promise
Whas' the reason for the emergency?

Manage expectations...

The busier your contact, the more in advance you need to communicate what they owe you (in meetings, feedback, etc.)

How to say no

voice concerns early
protect your team

"Bless and release" : fire bad client -- avoid toxicity

Know your modules: difference between 10 minutes and 10 hours for a feature.

Blaance development load against developers and non-developers for site configuration tasks : consistent method for initial site configuration

Work closely with IA/UE designers so they know what's possible with Drupal. Taxonomy and Views make functionality possible on Drupal sites that would be prohibitively expensive to build elsewhere. Likewise, know how to suggest features you're [familiar with] to clients.

Give Back (to the community) -- many ways to do so...

Selling Drupal Services (DrupalCon DC 2009 notes)

Jeff Schuler — Fri, 06 Mar 2009 17:44:53 +0000

Selling You and Your (Drupal) Services
Neil Giarratana

True genius is pulling threads out of the ether, and giving them reality. "Oh wow, nobody's ever said that?"

If you do the right thing, winning takes care of itself.

What you're selling is invisible.

The true product of a business is the business itself.

We talk about Drupal at the tail end of a meeting with a customer. Talk instead, about our business, our process, our team... You're really selling yourself. Create a "love mark" for your brand -- Apple, Starbucks. Don't create satisfied customers, create raving fans!

Your customer's view is the only one that matters. Customer is not too stupid to understand -- rather, you're too stupid to put it into words they can understand.

The way you deliver is at least as important as what you deliver. Even if you delivered a solution to spec, if you don't sell comfort, you haven't delivered.

Here is Lucidus' sales pitch for Drupal. First, they don't talk about Drupal. They don't use the word, Drupal. "For every site we build, we use a content management system..."

It's easy to install
It's easy to use
-- end customers shouldn't get Drupal (core) -- it's your job as a consultant to package it in such a way that it's easy for the client.
It has lots of features: point out a few key ones... 4,400 modules, (good reason they need you)
It's free
- Using FOSS, we are well positioned to take advantage of the financial situation. Longevity of project.

"No, really... why do you use Drupal?"

Configuration vs. Customization
- Drupal provides configuration rather than code for site building: New content types: Views + CCK solves so many problems without code.
What's a content type? Web pages, photos, books, blogs,...
Comments on anything
File attachments
Keyword tagging -- tying things together (related items...)
Subscriptions: users "ask" to be notified any time a produc of a certain type is added
Restricted access to anything... groups
Syndication (B2B)
Drupal was built for this -- for connecting things together.

Sales Process

Do you know how you get your leads?
Who are the targets? Define well.
Set up a campaign for those targets
Find partners: team up with someone who fills in your gaps
- Get in with a good marketing business: they'll send you business
Targets -> Campaigns -> Leads -> Opportunities -> Quotes -> Contracts

Lessons Learned

Lucidus has charged in 5 different ways, but now have two ways:
- build standard Drupal sites w stuff they've done before at a fixed price
  - The better they get, the more money they make
  - Charging hourly, the opposite happens
  - Customer gets pissed if scope expands, but if it takes less time, you don't make up the difference
- do blocks of hours, pre-paid in advance
  - here's our expectation, but this is all going to depend on what left turns you take... don't you want the possibility to take left turns?

Getting the Gig

No hocus pocus or slicky-boy sales guy: Here I am, here you are, let's talk about what we can do.
Listen! Listen! Listen!
- You're not allowed to tell anything about yourself for the first 20 minutes
- "Can we hear about you?"
  - "if you'd please indulge me, I'd like to know more about you so I can tell you the right things."
- People want you to feel you sympathize with them.
- Most important thing you can do is find Joy in their story.
- "I've got the greatest job in the world -- I get to help build all these businesses."
Find their pain (at a high level)
Explain your USP - Unique Sellling Propisition
Let them set the next step
- "What would you like to do now?"

Qualifier Questions

We do projects between $1 and $1000 range... Would you be in the $1-$100, $100-$200, $200-$500, or $500-$1000 ? (if it's not in your range, refer.)
What's your time frame to make a decision?
What is your launch date?

Closing Questions

On a scale of 0-10, (0 being not interested at all, 10 being ready to move forward,) where are you?
...How can I get you to 10?
How would you like me to proceed?

Contracts...

Set expectations for payment increments
Get a signature on everything you do
Get a blanket agreement

Q&A

"We're not going to respond to an RFP -- we find that the relationships we like to build are not built well based on RFPs, which tend to be $ focused, and not really tailored to building a solution that suits you. Very best of luck to you, I'm happy to have coffee sometime, and talk (no charge...)"
When do you make the commit?
- We do a one hour meeting, and they're going to start charging them after the first hour.
- Why don't we go through the spec process... If, at the end of it, you don't want us to do this, you can walk away with the spec and bring it to someone else...
Every site they build, they package in a support contract -- regardless.
You (client) want me to make a profit -- you want me to stay in business!

Great Books

Promiscuous Drupal: Building Your Site With Web APIs (DrupalCon DC 2009 notes)

Jeff Schuler — Fri, 06 Mar 2009 15:26:39 +0000

Promiscuous Drupal: Building Your Site With Web APIs
Jeff Eaton

How many folks in this room came to Drupal after trying to build their own CMS? Quite a few... Drupal tends to emphasize homegrown, native solutions to everything.

"Drupal: the blogging software with the worst forum software built-in." -- Walkah

We love Drupal's swiss army knife -ness, but if we don't keep our minds open, and stick to the Not Invented Here -- not use existing tools -- we end up with a humungous unwieldly mess.

Drupal as one tool among many.

Enlarging the Toolbox

Spam Filtering (Mollom, Akismet, etc.)
Searching/Indexing (Solr, Google, Acquia, etc.)
Feed Generation (Feedburner, etc.)
Content Tagging (Calais, TimesTag, etc.)
Authentication (OpenID, etc.)
Sending bulk mail (Mailchimp, etc.)

Whole category of modules for Third-party integration.

Farm out stuff that:

Benefits from centralization
Requires expertise
Demands massive processing or sample data

Taking it up a notch:

Posting links? Use Delicious
Shoutbox? Use Twitter
Photos and galleries? Flickr
Video? YouTube, Blip.tv
Reviewing books? Amazon, Goodreads
Comments? Disqus!
Blogging? Use WordPress!

You will never...

enter product information as accurately as Amazon!
be as popular as Flickr.
write a Firefox Plugin for Link Field
enjoy transcoding your own video!

To take the red pill... two approaches:

become a crossroads
enhance native content
- value add what you're already managing internally

When does this work?

Common on heavily social sites
Facebook
News sites
Personal Aggregators (FriendFeed, Tumblr)
External content is treated as something that lives out there

Users are active on your site even when they're not really doing something on your site.

How?

Aggregator module (themeable now)
Activity module (supports views!)
DayLife API, NowPublic's new service
Native modules (delicious, flickr, etc.)

One step further: using external sources as your primary source of content...

Does this make sense?

How hardcore are you?
How badly do your content producers hate the node form?
External content is treated as something that lives in here
Drupal becomes a presentation engine
Drupal becomes a ... mashup?

How the...?

Native modules
FeedAPI + Feed Element Mapper
Nodes become flexible wrappers
Lay your own pipes (we'll see this later)

Our stack (LAMP + Drupal) ... and 3rd party bits. Moving toward Drupal as one element in the rest of the services system.

Downsides

Filtering the firehose (tagging helps)
"Importing" the internet
8192 auth methods (OAuth helps?)
Even Amazon goes down... but flipside is that there are a whole lot of others tying their fate to same:
Magnolia ...

Congrats, you're no longer a module developer, you're an integration specialist!

Doing it with Drupal:

drupal_http_request()
PHP 5 - SimpleXML & JSON
drupal_write_record()
Expose your tables via Views
Wrap local vs remote
(See Building APIs That Rock)

If you release it

Support the full API.
Fail gracefully
Don't force data into nodes: store stuff in their own tables, and maintain data structure of original source as well as possible. For, say, Amazon integration, just store ISDN # in a node with a CCK field
Cache data whenever you can
Don't write an SQL query builder.. Integrate with Views
PHPDoc, PHPDoc, PHPDoc!

Don't forget

Respect API policies (Limits, Caching)
Don't trust the data you get back
There are cutting-edge options
- CustomViews queries (Flickr?)
- In Drupal 7, FieldAPI supports remote fields
- NowPublic's Aggregation system: building a Drupal site on a front end to back-end APIs/processing built in C++

Leveraging

Drupal is a great tool
It's strengths can be weaknesses
Outsourcing services
Drupal can be a mixer
If you roll your own, think ahead

CCK Mashup -- Oh The Things You Can Do! (DrupalCon DC 2009 notes)

Jeff Schuler — Fri, 06 Mar 2009 15:17:02 +0000

CCK Mashup -- Oh The Things You Can Do!
Karen Stevenson and Alex Barth from Development Seed

FeedAPI

Feed Element Mapper - create CCK fields in a FeedAPI content type, and map elements in a feed to those fields.

CSV Parser - use FeedAPI to parse CSV

Also exist mappers for M-Fields, mappers for Taxonomy, mappers for less common CCK fields (email, link)

Can specify a field as GUID to compare for updates

For other XML formats? Write your own parser: not too hard with PHP5's SimpleXML

Prepopulate - Prepopulate fields when creating a new node by specifying in URL

Drush - command line Drupal productivity (DrupalCon DC 2009 notes)

Jeff Schuler — Thu, 05 Mar 2009 22:59:42 +0000

Drush - command line Drupal productivity
Moshe Weitzman and Owen Barton

Drush module: why?
- quit switching between browser and shell -- stay on the command line
download and put it anywhere
- can symlink or alias to run it
doesn't matter, D5, D6
- for more specifics, get Drush extras module
can specify directories and URI, or it'll try to figure out based on working directory

commands... try drush on the command-line to see!

Front/Back End Performance – Make Your Website Lightning Fast (DrupalCon DC 2009 notes)

Jeff Schuler — Thu, 05 Mar 2009 20:10:28 +0000

Front/Back End Performance – Make Your Website Lightning Fast
Konstantin Käfer (Front End) and Khalid Baheyeldin (Back End)

Front End Performance

how user perceives?
how fast website is loaded?
how fast the page loads?
how well the page performs after it's loaded?

styles, scripts, images account for 80% of load time

background images, images, other media, html, css, scripts, other media...

over time:

HTML (generating page in Drupal)
CSS
Scripts
Backgrounds
Images & Media

Tools & Measurement

Measure: to prove optimization success

Overall load time?
Page size? Not a perfect metric: ping time, roundtrip time
Time until DOM is loaded?
Time until page is functional? : users perceive differently, or use different things

Tools

Firebug's Net panel
YSlow
AOL Page Test
IBM Page Detailer
Pingdom
WebKit's Web Inspector
Web Debugging Proxies: http://charlesproxy.com, http://fiddlertool.com

Most display as waterfall: Start, Connect, First Byte, Last Byte

How can we optimize?

Reduce requests (More requests more troublesome than larger files)
- CSS Sprites
  - many images into one file, and shift into view using background-position
- Aggregate scripts and styles
  - built into Drupal
  - Sophisticated: Support File Cache module
- Don't use redirects
Caching
Use CSS instead of images
- -moz-border-radius, -webkit-border-radius, border-radius
data: URLs in style sheets
Use a CDN
Caching
Gzip
CSS at the top
- Browsers don't render until all CSS has been loaded
scripts at the bottom
minify CSS and JS
remove comments and whitespace
- still savings, even with GZip
- Drupal's aggregator or sf_cache.module
Parallelization
- 2 requests per host name in parallel
- use multiple host names: higher parallelization
  - but don't overdo it: DNS lookups will slow down
- browser support for parallelization
reduce image weight
- OptiPNG, PNGCrush
  - Removes invisible content
  - Lossless recompression
- JPEGtran/ImageMagick
  - Remove color profiles, meta data, ...
  - Lossless
lazy initialization
microoptimization
- only if you have optimized everything else
  - "Premature optimization is the root of all evil." -- Knuth

Back-End Optimization

definitions

Performance
Scalability
High Availability
Load Balancing
Performance Assessment/Analysis
Performance Optimization/Tuning

Define goals and objectives first

Optimization: diminishing returns

usually low-hanging fruit with dramatic performance; after that, it's more expensive and/or more difficult
infrastructure, patching Drupal, re-architecting the app

proper diagnosis is essential for any solution; otherwise, running blind
must be based on proper data, and analysis of the data collected

validate results on a test server

Physical server matters

dedicated / VPS
Multiple cores are the norm, now: 8 better than 4 better than 2
Lots of RAM

Multiple servers

one DB server, multiple web servers
Can use DNS round robin for load leveling, or proper load balancers (commercial / free,) or even a reverse proxy
- only if you have the budget
- complexity is expensive -- running cost
- tuning a system can avoid or delay the split

LAMP Stack

Linux

Use a stable distro: Debian stable, Ubuntu Server LTS, CentOS)
Use recent versions
Use whatever distro your staff has expertise in
Be a minimalist; avoid bloat
- Install only what you need
Balance: "compile your own" vs. upgrades
Compile your own:
- pros: full control on specific versions
- cons: not easy to do security upgrades
Apache
- mod_php and mod_rewrite only, from the start
- MaxClients:
  - too low: can't serve a spike
  - too high: mem can't keep up with load & start swapping
- MaxRequestsPerChild
  - to terminate process faster and free up memory
- KeepAlive
  - should be low: ~3 seconds
- mod_gzip/deflate
lighthttp (lighty)
- popular with RonR crowd
- 1MB per process
reports of bad mem leaks, recently
nginx
- newcomer
- more stable than lighty

MySQL

MyIsm
- faster for reads, less overhead, poor concurrency
innoDB
- transactional, slower in certain cases (SELECT COUNT(*))
- Better concurrency: good for heavily hit tables, such as sessinos, watchdog, ..\
Tuning
- Query Cache (most important)
- Table Cache
- Key buffer
- InnoDB for sessions, watchdog, accesslog
- Temp tables on Linux tempfs (in memory)
Replication
- patch needed until Drupal 7

PHP

Always use recent version: 5.2 min for 7.x and many 6.x contribs
Install an Op-code cache
- eAccelerator (fastest), APC (most stable)
- Benefits
  - Dramatic speedup of apps,
  - decrease in CPU/mem utilization
- Won't help for
  - not just code execution
  - network connections
  - sorting of arrays
  - heavy database access
  - tagadelic, node access modules, admin_menu, forum, tracker
mod_php
- normal apache config
- shared nothing
- no state retained between requests, less issues, most tested/supported
- stay with mod_php if possible
php as CGI
FCGI
- use if possible
other exotic ways...

Drupal

db intensive
disable modules you don't need
make sure cron runs regularly
enable throttle
- be wary about throttle and cache
does your module do stuff over the network?
- cache the data, and use job_queue or queue_mail_module
Drupal Caching
- anonymous only
- enable page caching: expire too often on a busy site, causing slow-downs
- set expiry
Boost for anonymous visits
Pluggable caching -- custom caching
Block caching -- core in D6
memcached
- necessity for every big site
- distributed object caching in memory
- no disk I/O (database or files)
- requires patches...
Advanced Cache Module
Slow modules
- Statistics module
  - adds more queries
  - disable popular content block
- Aggregateor2 abandoned, thank god
- Tagadelic
- Admin_menu (adds up to 500ms)
- Node access modules with large number of nodes (10,000 nodes)

How to measure

top
htop
atop
vmstat : mem and swapping: can show history (30 minutes ago..)
apachetop
- reads and analyzes Apache's log
- how many requests/sec
mtop, ntop
MySQL report, query timer, slow query log
Stress Testing
- ab, ab2, Seige, JMeter
Nagios monitoring: now Drupal Nagios module
analytics, AWStats
Performance Logging

Building a Frankenstein monster & how to maintain it (DrupalCon DC 2009 notes)

Jeff Schuler — Thu, 05 Mar 2009 18:51:20 +0000

Building a Frankenstein monster & how to maintain it
morton.dk

Setup: ideal

clients give you enough time to get the job done
the clients don't bitch about small stuff, and pays in advance
wire framing is complete! "nothing's gonna change, ever!"

Setup: reality

limited time theming is the last in line: after design & functionality
the client can't understand the site before it's completely done
wire framing missing for 25% of the site

Designer ideal:

understands Drupal
- knows limits and not gonna bitch about them
- uses a grid system

Designer reality:

don't care about mighty Drupal
Have no clue of limits
Don't use grids: "I don't do boxes! I'm an artist"

Developer: ideal:

all output is overwritable through theme();
not have lots of CSS files

Reality:

hardcoded HTML
$styles full with css files from a ton of modules - with a ton of classes

Editor ideal

only adds words & predefined images to the content
kindly ask to have changes in the css if something's gotta change
knows that inline styles are made by satan

Editor reality

copy-paste directly from word
throw in classes to the style.css
adds inline styles anyway

Themer ideal

knows every corner of theme()
works hard and long and listens to the codes, pm's & designers
always makes the wise choice

Themer reality

don't know every corner of theme()
uses {display: none}
+10% hotter than the rest - that's just a fact

To control the monster: the mothership:
Basic starting theme & subtheming

In the mothership theme:

mothership.info
page.tpl.php
node.tpl.php
style.css
template.php

Keep mothership as theme as possible.

Subtheme:

subtheme.info
page.tpl.php
node.tpl.php
style.css
template.php

overrides in subtheme for page.pl.php, node.tpl.php

Note: if you need a node-foo.tpl.php as a subtheme override, you need a node.tpl.php in your subtheme, too.

mothership: only the most common elements; as little as possible

subthemes: everything else

files

mothership.info includes meta, regions, features, stylesheets, conditional styles, scripts
subtheme.info: adds "base theme", stylesheets
mother/style/reset.css: clean up browser hell, resent font size, remove img borders, .clearfix, clear-break, ... these can be found online
mother/style/drupal.css: makes block admin workable, menus, tabs, drupal stuff...
mother/style/grid.css: "width sized class definitions", grid-1 {width: 60px}; grid-2{width:140}...
- google: mark boulton & grids
- 960.gs
- blueprintcss.org
mother/style.css: basic layout, design, colors...
style/grid.css: overrides
style/styles.css: overrides
styles.css: quickfixes, clean it up every so often, move the good stuff up into style/style.css

StyleStripper module

selectively remove all unwanted .css files from theme (from other modules, core, etc.)
different settings for different themes

References

A Paradigm for Reusable Drupal Features (DrupalCon 2009 notes)

Jeff Schuler — Thu, 05 Mar 2009 16:38:43 +0000

A Paradigm for Reusable Drupal Features
Young Hahn, Robert Soden, Ian Ward (DevelopmentSeed)

In theory: the problem + solution
In practice: feature building demo
The future: what comes next

A blog is a "feature". So is a pressroom, an image gallery, a publication library, etc., etc. Blog can be created using core module, or cusotm recreated.

Example: making a pressroom: content types, fields, views, etc.

3 problems: slow, not reusable, disconnected pieces

A (turnkey)feature module: takes everything usually done in UI and puts it into code. How?

Scripted: content type, fields, taxonomy, menus
Exported: imagecache presets, block visibility, block views, page views, comp. relations

Spaces + Context demo

Code:

is reusable
is distributable
can be kept under version control

Getting site-building into code:

Scripts:

pain to debug
pain to maintain
not easily extensible

Earl Miles' Chaos tool suite

Exportables:

abstract data storage so that your module can be indifferent to whether it's in code or db
provide definitions, not scripts
are easy to maintain
are overridable and extensible

Modules with exportables?

views
panels
context_ui
imagecache

Recipe for exportables

string identifiers
API functions for collecting all items
Structured array or object to represent your item

Future: ability to export features into module form

Developers: move toward adding exportability into your module!

Optimizing your LAMP stack for Drupal (DrupalCon 2009 notes)

Jeff Schuler — Wed, 04 Mar 2009 21:23:51 +0000

Performance and Scalability: Optimizing your LAMP stack for Drupal
Eric Mandel

Performance

different things to different people
amount of useful work accomplished by system, compared to resources used
however you define the atomic unit of measurement -- that one thing -- it's generally how fast can you do it.
- in a web app, how fast can you serve something (a page) to a user?

Scalability

Can the system grow without degradation
How big is the system and how big can you make it?
- In a web app, how many users can your app serve while maintaining acceptable performance levels?

Performance: 1 user, Scalability: manu users

Drupal's Performance Problems

Lots of queries
Unoptimized queries
search gets slower as nodes increase
3rd party modules are performance unknowns
logged-in users require more queries
handling millions of nodes requires hacking core

[Damn... just lost my notes here...]

Scalability Prerequisites (cont.)

Isolate multimedia files
Use a CDN or have a plan to use a CDN

OpenID, Drupal and the Open Web, (DrupalCon 2009 notes)

Jeff Schuler — Wed, 04 Mar 2009 21:01:24 +0000

OpenID, Drupal and the Open Web
James Walker

He's given this talk before, but all of a sudden (months) things have gotten more interesting

today: "web 2.0"
All of these various sites where we need to recreate identity and relationships

password antipattern
giving out passwords for other sites -- you've given facebook access to all of your Gmail, Google Docs, etc?!

how about User Controlled Authorization
I am the center of my internet
It's MY identity, they're MY contacts, it's MY content

Facebook Connect

You are not Facebook!

USER-centric,

DiSo: Distributed Social
activity streams, microformats, portable contacts

OAuth

realms
per-site
OAuth in D7? (because Dries mentioned it...)

OpenID

Identity: need unique, ubiquitous identity globally (on the internet)
what about when someone dies?
what about single sign-out
how to merge OpenIDs or Facebook Connect IDs with current IDs on sites
Who is the "you" that's in the middle of the graph?
Are you a URL? An email address?
- globally unique
Do you know you have an OpenID (if you have a Google, Yahoo, MySpace, AOL, etc., account) ?
Usability issue: redirecting to OpenID provider's site on login. Facebook Connect does this nicely.
Facebook is showing a certain amount of recognition of OpenID: joining board as corporate member, hosting UX conference to share what they learned through Facebook Connect...
What about using DNS's distributed model

So, OpenID and Drupal

Drupal's tagline: community plumbing... new one: social publishing?
plaxo + Google teamed up to do a Facebook Connect with open standards: 92% success rate!
http://drupal.org/project/openid (in D6 core now)
http://drupal.org/project/oauth
http://groups.drupal.org/openid

Drupal Module Development Kickstart (DrupalCon 2009 notes)

Jeff Schuler — Wed, 04 Mar 2009 20:08:39 +0000

Developing Modules kick-Start
Gábor Hojtsy and Peter Wolanin

This is about developing modules, but if you can find a module that's already out there and being developed, use it!

if you can avoid custom code or modifying, build the right way and use existing code when available

Change almost anything without hacking core.

What is a module?

.module + .info text files
Implements hooks, Drupal exposed interfaces to plug into the system

replace "hook" keyword with module name

to implement hook_perm(), write modulename_perm()

module directory goes in:

sites/all/modules/modulename (or)
sites/domainname/modules/modulename

example: how to allow admins to delete something faster

add a 'delete this post' button to a node's menu
hook_link()

example: focused search

add a radio button to search panel to search all content or search a particular content type only
use hook_form_alter()
gives modules a chance to change forms before they're presented

example: editing the Contact form

hook_form_alter() again

example: module from scratch: whiteboard (from Szeged)

hook_menu()
- make page callback to custom function (not a hook)
- access arguments
but avoid this! use Views

module_invoke_all()

"if any other module is interested, take action now"

References

Developing for Drupal
API Reference
or run API docs locally with API module
#drupal IRC channel
books
Drupal and CVS
http://donthackcore.com

Totally Rocking Your Drupal Development Environment (DrupalCon 2009 Notes)

Jeff Schuler — Wed, 04 Mar 2009 16:53:38 +0000

25 Tips for Totally Rocking Your Drupal Development Environment
Sacha Chua

Full Session Notes on Sacha's blog

Browser tips and plugins (firefox)

Firebug
firefox -ProfileManager -no-remote
Tamper Data
iMacros
Drupal for Firebug

Drupal modules

Admin Menu
Devel
- check out SMTP facade tool
SimpleTest
Login Toboggan
XDebug

Source code and configuration management

Check in your entire source tree
Organize your source code
Check-in clean source for third-party modules
Test all upgrades against a production copy
Manage your branches and merge changes

Automation and regression testing

Use build tools
- make, ant...
- define a common set of commands, and have config/setup variables on the back-end
- allow yourself to forget SQL commands, URLs, passwords...
- Makefile targets:
  make mysql     connect to the database
  make backup     make a copy of the database
  make backup-partial     make a partial copy of the database, omitting accesslog data and other unnecessary information
  make restore     restore the database from the backup
  make dev-restore     restore the database from the backup, and enable all the development modules
  make clearcache     restore the database from the backup
  make tags     restore the database from the backup
  make doc     restore the database from the backup
  make test     run all my project-specific tests
Learn or make tools to save yourself work
Drush - the Drupal Shell
Simpletest
test code AND webpages

From the audience

Vimperator
Virtual Document Root: svn checkout in a particular directory
geben: XDebug with emacs
unfuddle hosted svn and git SCM
Trace module: trace hooks and timing
groups.drupal.org/Drubuntu
Coder module: Developer Module that assists with code review and version upgrade.
Sun NetBeans
CustomError for handling 403s/404s (as LoginToboggan alternative)
Creating aliases for drush for different projects?
Drush will become less of a module...
make sure development modules don't end up on live sites!
use sites/localhost for development modules
insert notification to warn you if they are installed
install profiles
for searching modules: don't google! check out all of contrib and do a recursive grep across it.
aegir - hostmaster evolved

State of Drupal (DrupalCon 2009 notes)

Jeff Schuler — Wed, 04 Mar 2009 16:13:56 +0000

State of Drupal
Dries Buytaert

talk theme: "Drupal Rocks"

Meet Dries in 1999 ~10 years ago
young geek: Assembler, Compilers, Linux kernel wireless, chess, stamps

Drop.org message board

Drupal 1.0.0 -- January 15th, 2001
for news-driven community or portal site

Jeremy @ kerneltrap needed a site to resist slashdotting
Dries gives him admin privileges to drop.org to try out

folks in Drupal community have tended to be young

Mark Boulton's redesign of Drupal.org
drupal.com forthcoming
Lullabot podcast an important milestone

action/attendance/attention doubling by year

downloads
conference attendance
contributed projects
uniques/month

Hundreds of thousands of websites
thousands of developers
more than 4000 extension modules
100% yearly growth
24/7
Free

Built by everyone
Controlled by nobody
Like the internet: always a little bit broken, yet works

"Replace planning with coordination." -- Clay Shirky

(Dries:)
"I never believed in formal roadmaps"
"I never believed in hierarchies"
"I never believed that the Drupal Association should be in charge of the roadmap"

Long tail of contributions -- difference between companies and open source

"How do we avoid creating institutions, how do we avoid doing planning?"

code freeze for Drupal 7 on Sept 1st, 2009.
doesn't mean it will be released then: will be released when it's ready

Drupal 5 had installer
Drupal 6 made it clean

Usability...

Drupal 7:

default install profile
usability team
Better media/document handling: File API
Fields API (CCK) in core
RIP Profile module
testing
serious database abstraction layer improvement

Dries would like to see these features in core:

OAuth Support
Job queues
RDFa output
Real-time messaging (XMPP)
Activity log

Start thinking of the internet as one big machine -- all devices are views into the one machine

Trends:

Linking machines (internet)
Linking pages (web)
Linking data (semantic web?)
Linking things (internet of things)

Linking data is called:

Kevin Kelly: "One machine"
Tim Berners-Lee: "Giant Global Graph"
"Web 3.0"

Drupal is a Movement

A movement needs a mission.
Right now mini mission is usability: allow folks to make any website they want easily, without restriction.
A larger mission is in helping transform the web.
There is a difference between telling people what to do and creating a movement.
A movement needs a healthy ecosystem.
A movement needs leaders.
Most people are not leaders, they're lurkers -- waiting, watching what happens.
You are not most people.

Building APIs that Rock (DrupalCon 2009 notes)

Jeff Schuler — Wed, 04 Mar 2009 14:35:39 +0000

Building APIs that Rock

Jeff Eaton
9:00 AM, 2009-03-04

Official session notes

Why APIs?

let other code use your features
- Focusing on writing modules that provide a nice interface to code as well as humans
- Examples of modules with nice APIs: Views, OG, Flag
hide complexity
- Better to write modules that use underlying functions than "pretending" to be a user and using forms
- hides whether ImageMagick or GD tools (or other lib) is being used
- provides "rotate" "scale" "crop" Messaging
- abstracts /how/ messages are being sent
- cache_get, cache_set (inside Drupal)
- provide simple interface to hide method underneath (memcache, etc)
Agreement about how things get done
- Token

Example: Creating VotingAPI

started by tearing apart someone else's module
find nouns: votes
find verbs: get / set

Example: myapi_do_something()

don't use enums and hard-coded numbers unless you know it's something that won't change
- use strings to allow others to make their own new type
well placed hooks allow other people to do the hard work for you!
module_invoke_all()
drupal_alter(): "it's like passing around the doobie of data" : everybody gets a chance. (you don't have to...)"
hook_form_alter()
hook_menu_alter()
who wants to hijack this before it hits the page?
let other modules have a chance to change or add more data
"template" design pattern
"if you build it they will hook"
layer verbs to hide complexity
account for new users

7 deadly sins

Lone Ranger (reinvents wheel, ignores conventions, ignores related apis... common to newcomers
invisible Assumptions (assuming the global $user, checking arg(0), assuming logged-in users... common to UI-centric modules
Helping them to Death ("while i'm at it, I'll set the breadcrumb, send an email, and add links, and... common to bloated modules
Leaky Abstraction ("just use this function... and learn everything that's underneath it! ... common in complex problem domains. Simplify! Assume sensible defaults, and get out of the way when appropriate
Fake Flexibility (Providing hooks that don't really provide the functionality they portray
Mission Creep (even worse than feature creep .. common with "solve a big problem APIs. Split it into smaller APIs!
Dependency Soup (Flip-side of Mission Creep. Focus focus focus! Make dependencies optional, pray for a package manager in Drupal

Rules of Thumb

cooperate
don't assume too much
do't be pushy
simplify or step aside
test your extensions
stay focused