Archive for the ‘Notes’ Category

A Firehose.

Wednesday, April 21st, 2010

The real-time web loves the word firehose.  I’m not completely sure where the word started being used in relation to streaming data, but the first time I cared was with Twitter.  The Twitter firehose, all the updates from everyone in real-time, the full stream.  It seemed like everyone wanted it, few people needed it, and even fewer got it.  Some every interesting things were built and bought (summize / now search.twitter.com)  Now every company offers a firehose, right?  Well some do.  Wordpress pushes all their updates in realtime via xmpp or pubsubhubbub.  And you can even purchase the full feed.  Others have stepped forward with streams of data, Myspace for example.

So what exactly is a firehose.  It goes beyond being a real-time push stream of data.  If you just had my twitter stream in real-time would you call that a firehose, hardly.  Firehose is all public data, the full thing.

As the Twitter Api says:

Returns all public statuses. The Firehose is not a generally available resource. Few applications require this level of access.

Now I’m starting to see more people say “we got a firehose!” when they dont have anything close.

First, Yahoo. See the Read Write Web Article (which shows they didnt look or didnt care, just repeat the Yahoo Press release)

The first sentence makes it clear that it is not a firehose:

The Updates Firehose API gives developers access to the full, real-time Yahoo! Updates index and allows developers to search, filter and combine Updates data using YQL.

That’s right, the full firehose to query.  What?  It isnt push, it isnt a full stream of data, it isnt even close.  Instead this is just a combination of several APIs from yahoo into one endpoint.  That’s cool for what that is worth, but that isnt real-time and it isnt streaming.

Second Spinn3r.  From their site:

Spinn3r listens to a new Twitter firehose API which is a sample of the full Twitter feed.

Again, if it isnt the full stream it isnt the firehose.  That’s like firefighters showing up with a garden hose. (likely the api method they use)

I’m glad to see that we have gotten more streams of real-time data on the web.  I’m glad to see some companies providing a firehose of their content, even if they charge for it.  The ability to build  some of the next large scale ideas depends on that.  Just dont crowd the search results with overusing the term, use it correctly.

Domain names + Unicode = Punycode

Tuesday, April 13th, 2010

I Saw.

I saw a unicode symbol of a white star enclosed in a black circle the other day, in a domain name.  So last night I set out to see unicode characters in a domain name for myself.  I started at my favorite Registrar name.com.  Some of the country code domains they sell allow unicode characters.

I Bought.

So I chose ⤷.ws  the .ws was the cheapest option coming in at $14.5o, and as you no doubt know .ws is the TLD for Government of Samoa.  First thing that I notice while setting up my DNS was that domain showed up as “xn--2ri.ws”.  What is that all about?  Well, although I really kinda hoped that the internet had progressed to the point where we can all handle unicode, it hasn’t.  So in order to implement Internationalized Domain Names or IDNs, we still use Ascii.  How?

I Wikipedia’d.

Punycode is a means to represent a larger character set (unicode) with a smaller more restrictive character set (ascii).  Do to some security concerns around spoofing urls many browser will show the punycode that represents the unicode domain in the address bar, although some do this selectively such as safari.  Many countries are retrictive on the allowed characters, usually ascii + their countries language, but there are some that have just allowed them all (thanks Samoa).  So my new url shorten is in place, http://⤷.ws

NMTechCal.com – Concept.

Wednesday, April 7th, 2010


The idea is for a central location to look for information about upcoming tech/web/design events in New Mexico (start with Santa Fe and ABQ).

  • Shared google calendar with event organizers. we all have access to enter our own info. (built in iCal feeds etc.)
  • Blog which we highlight new, and upcoming events. This is authored by event organizers, and other contributors.
  • That’s it. It think simplicity is key. But even this small step would be a big help.

If you know a group that would like to post their info, or if you would like to help by contributing to the blog, email me: mick@davidmichaelthompson.com

Calling all of New Mexico’s Tech Community

Wednesday, April 7th, 2010

About a year and half ago I traveled to Boulder, Co for the first time.  What I found there was a vibrant web developemnt / design community.  Here is what I mean by that.

Tech Community

In Boulder: At least a couple of events (great then 50 people) a week.  It isnt all the same people at all the events, although at the major ones you can expect the regulars, but in general a good diversity and networking / learning opportunity.  Some of these events are fun geek nights out (Ignite) some have a bit more structure in content (Boulder New Tech Meetup, generally has some demos / presentation about companies and products), some are just user groups exchanging knowledge.

Along with all these events comes many organizers.  These are people that want to give back to the community, want to see it grow, and want to be a part of it.  In Boulder I have seen many people reach out to be a part of this, and I’m starting to see more of that here in New Mexico (read as: a very good thing).

In The Land of Enchantment

Over the last year I think we have seen a good increase in geeky events in New Mexico.  NMTC, Barcamp, Webuquerque, IgniteNM, AIGA, SynerQue, ABQ Web Geeks, Several User Groups(ruby, python?), as well as others I’m sure.  One thing that I think we lack a bit, is cohesion.  Just last night I heard about 2 events that I would have been interested in attending, however I didnt hear about them in time to attend.  Part of this is trying to keep up with multiple groups and just getting lazy about doing so.  Some of it is not knowing about new groups that are just getting started.

The Solution?

There is a question mark, because this is still very open for discussion.

The first step that I’m going to take (and hopefully getting some others involved too) is to start a calendar of all NM Tech Events.  This will be administered by all the group organizers.  Have a group or are you planning an event? Contact me (mick@davidmichaelthompson.com) and I’ll set you up with access to the calendar.  This is based on a shared google calendar, and a website to display the events, and highlight some.

Step 2. Blog covering upcoming events, highlight recaps of past events, and discuss new or current groups.  I’m willing to do some of the writing, but I really want this to be a group effort (i.e. I’m looking for a bunch of other contributors)  This will all at nmtechal.com If you are interested in contributing in any way, let me know.  I want to run this very open to the community at large, everyone welcome!

Step 3. Talking with Chad Kieffer, he has the idea for a event where we actually all pull out our laptops and work on some code.  Very hands on.  I’m thinking about a “Web Project Night” at the Santa Fe Complex.  It would meet 1-2 times a month, we’d decide on some sort of small project, and try to use new tools to accomplish the goal.  Getting to work with others, build something, and try out some new things along the way.

That’s it from me.  Grow community, Grow!

Hiring

Wednesday, June 10th, 2009

I’ve never really hired someone before.  I’ve worked with people, perhaps I’ve even had some input on a job candidate, but I have never hired anyone before.  I’m not exactly hire now, but I am looking to sub contact out some of my web development work on a regular basis.  So I put up a craigslist ad to see what kind of people are out there and available.  In the Santa Fe area I dont think there is a huge selection of developers, but there might be some.  So far I have gotten 3 responses(The ad has been online for about 5hrs)  This should be an interesting adventure.  We’ll see how this turns out.

WordPress Page Titles Vs. Menu Text

Wednesday, March 4th, 2009

This came to my attention on Twitter yesterday thanks to @idesignstudios. The question is, how do you have a the link text and the page title different?  Perhaps the In the Menu it should say “About” but the Page title should say more, perhaps “Read All about our Company”.  You dont want the long title in the menu.  How can this be done in wordpress?

Selene (@idesignstudios)  found an answer to this here(cssglobe.com). The solution works, but it involves modifying core wordpress files.  Which I would strongly recommend avoiding.  Wordpress, as everyone knows, tends to release a lot of updates (Yes I have run into plenty of update fatigue myself). So dont make your update harder.  When you upgrade, all of the core files will be overwritten, and you will have to remember to apply the patched code again. Perhaps having to lookup the article again, find the right file, find the right line… Can you see that might be messy 6 months or a year after doing it the first time?

But I’m not here to complain.  In fact the code and idea on cssglobe.com handle the problem very well.  However let’s move that into a plugin so when we upgrade the code our changes wont be overwritten. Here is my plugin for just that: Page_Title_Changer.zip

Here is an example:

 With the plugin a metabox is added to the right hand side of the page edit/add admin screen, From there you can set the text for the page title (in the example: “Let us know what we can do for you”).  This is saved as metadata for the page, and a title filter handles choosing the correct title, by location.

I took 20mins to throw this together, so if you find something that doesnt work out as it should, send me an email, I’ll take a look.

How do you interact with your clients? – Issue Tracking

Monday, December 29th, 2008

Ok, I’m going do a series of this posts on “How do you interact with your clients?”, I’ll share what I do, what I like about it, and what I dont, and maybe some new tools to try.

In any project issue tracking becomes needed. You might be working alone on a project for a client, and they find things in the demo that they would like changed, those are issues.  You might be working on a large project with a team, and have QA personal when they find bugs, those are issues.  When the team or person working on the UX doesnt like the interaction, that is an issue.  Or maybe you just find an issue all on your own (maybe a few issues) track those!  Notes about resolutions can be useful, even more so when you are working on 5+ projects.

Where to track?  

Bugzilla – Open source, the de-facto bug tracking for open source projects, well at least at mozilla the creators of bugzilla as well as many other open source projects. Though I have seen less and less projects using Bugzilla lately.   Pros:  it can do just about anything, a million features, easy enough to add anything else you need / want.  Cons: you have to setup, you have to host it, too complex for most clients to use, too clicky. Update: look like there is a new version of bugzilla, 3.2.  I havent tried this one yet, but it does promise a greatly improved UI.  I hope so.

Email – I have done this.  And it can work with limited success.  It is better then nothing, but at the same time has a lot more time involved.  At least you get a history, though it might be in a less query-able format.  This seems like the place many projects start.  But as the spreadsheets that you start emailing back a forth get larger, this becomes far to hard to keep up to date.  Pros: easy to setup, everyone already knows how to use it. Cons: data will be out of sync quickly, it is harder to search and query, i.e.  what are my current open issues?

Basecamp – This is a nice project management tool.  Great interface, excellent workflow, and a simple set of features that is everything you need.  Issue track is missing a couple of things here though.  Mostly because Basecamp tracks todos.  You lose source control integration, issue states / statuses.  But I find that for many clients this is the simplest and easiest to use.  This is often the best choice for client interaction,  It sure beats email.  Pros: easy to learn, most people will understand right away.  Cons:  it might not have every feature you are looking for in issue tracking.

Lighthouseapp.com – These guys are newish, but have created issue tracking that makes sense.  It doesnt get cluttered by a million features that are never used, but it still has the features that count.  I’m using this for my startup.  We can track milestones, issues that need to be completed for each milestone, general issues, and even create some wiki like pages for easy information / idea sharing.  Their interface has a very good feel, easy to use even for your first time, and you are never left guessing how to do something, it is obvious.  Pros: great interface / UX for issue tracking.  easy to learn.  Cons: Might be more than a client wants to deal with.

Which method of issue tracking to use?

That depends on you and your client or team.  For a development team I heartily recommend lighthouseapp.com.  If you have a lot of client interaction in your issue tracking, it might be better to use basecamp.

Anyone like something else?  I know there are a lot of companies in this area and some really great services, so what else I’m I missing? 

 

Filter HTML

Wednesday, October 1st, 2008

Many websites take input from users.  If your website is going to take that input a redisplay it somewhere on the site, you really need to filter your html.  If you are lucky the requirements for your site will let you strip out all html code, if your not lucky you will have to filter it.  

Why Filter

First, why accept html at all?  because it is easy enough for users to work with, for a WYSIWYG editor to work with, and it is what you are going to be displaying, really it is best all around in terms of your site’s performance and your user experience.  

Ok, so why filter?  Well there are some hurtful people out there, and some people that dont know what they are doing, both can make you look bad.  First the people that dont know what they are doing will forget to close their tags(leaving a <b> tag so the rest of the page can be bold), and using other markup that will ruin your design.  Then the hurtful people will come in a add some javascript to destructive ends, compromising the security of your site.  And keep in mind that script doesnt have to live inside a <script> tag.  it can be in many attributes, such as onmouseover for example. combine that with some inline style that enlarges and positions text to cover the whole page, and boom!  the hacker just got their malicious script to run on your site without a script tag, not good for your users.

How to Filter

We understand the need, now how do we accomplish the task?  3 main points.

1) Whitelist tags and attributes.  Create a whitelist of allowed tags, and their allowed attributes.  Whitelists are better then blacklisting.  Cause they should be shorter,  they are easier to maintain, and more restrictive.  A comprehensive blacklist could take a long time to make, and whenever a browsers decide to add support for new tags, your blacklist requires updating.  If you use a whitelist, it is shorter, and wont break as new tags are supported.

2) Balance is needed.  Your page can be ruined if the user submitted code includes some </div></div>  Or what if the user opens a tag that they never close… maybe  <center>  What will your site look like then?  You need to add balance to your user submitted html.  Balance all tags.  Also keep in mind tags that self close, <img> or <br> for example.  and to be XHTML compliant make sure they include the self close <br />

3)  Proper Nesting.  Improper nesting in certain browsers can lead to trouble similar that that of unbalanced tags.  Check for <b><u>text</b></u>  

Here is the Code

So enough talk here is some links to helpful code to get this done:

Also check out a project called Tidy, it has a lot of this functionality built in, and is available for many languages.  - HTML Tidy Project Page

So long, and be safe…Filter your Html.

OPML

Sunday, May 25th, 2008

opml.gif OPML, the feed of feeds.  Really.  I recently got a chance to work with opml, and I was bored on a sunday night, so I thought I would share with you a couple of thing I observed. 

First off the link:

opml and asp.net – That helped with parsing the opml file and getting a dataset from it. 

OPML files for the most part have a simple layout.  <outline> tags that hold attributes for the feeds type, xml link, site link, and title.  You can add a couple more things, but really that is basically what you will see in every opml file.  

According to opml.org the goal of opml is:

 The purpose of this format is to provide a way to exchange information between outliners and Internet services that can be browsed or controlled through an outliner.

OPML is also the file format for an outliner application, which is why OPML files may contain information about the size, position and expansion state of the window the outline is displayed in. 

I’m not really sure what they mean by that, but I do know that opml is the way to import / export / transfer subscription lists.  And since the web is crazy with subscriptions these days, opml is found in most feed readers.  So if you are writing a feed reader, you should have it on your feature list.

Well opml is really no more complex then that.  So I’m going to get back to coding…

Notes: installing Django / Python / Mysql on OS X

Monday, February 18th, 2008

Trying to branch out in my scope of web development, being mostly a PHP and .NET developer, I decided to try Python. Now I still do not know much about Python development, packages, or even the syntax. Really I am just getting started with it. But I thought I would link to a couple of articles that helped me get setup to develop under OS X for anyone else starting out.

Why try Python?

I started watching some interviews on the yahoo developer network. One in that caught my attention was with Matt Mullenweg, founder of Automattic and developer of wordpress. When asked the question, if you were to start out on a project the size of wordpress today, is there any other language that you would use/consider? And sure enough he answered Python. For the pass couple of years I have thought about using python, and I even once installed it. But that has been as far as I have gotten.

I also started using the website Pownce.com recently. Pownce is a social network allowing you to send messages, files, links, images and video to your friends easily. Well, Pownce who just opened up their site to the public (they where in a private beta for about 6 months) uses the django framework for python for the whole site.

Last Friday I watched a video by Jacob Kaplan-Moss one the the lead developers of django, and found some the of the concepts really interesting. Django was and is developed out of Lawerance Kansas, you know the center of web development. It was built by the web team for the local newspaper in order to better mange the many sites they host and their constantly changing needs. Jacob explains that better than me, if you watch the video. I downloaded the Framework on Friday night (as I was driving back from Kansas City Saturday and Sunday) as something to look at on the way.

Getting Started

Well fortunately Python is included in Leopard, so I didn’t have to download that over my gprs connection in Kansas. Django is very easy to setup, basically run a script from inside the extracted django directory.

sudo python setup.py installOk with that installed where do you start? Django provides a nice walkthrough of setting up a project, what files are included and a step through of the built in features. After following about a page of the walkthrough I got to the point of being ready to sync my DB code with the real database, MySQL.

python manage.py syncdb

You then get this:

python manage.py syncdb
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/Library/Python/2.5/site-packages/django/core/management.py", line 1672, in execute_manager
execute_from_command_line(action_mapping, argv)
File "/Library/Python/2.5/site-packages/django/core/management.py", line 1571, in execute_from_command_line
action_mapping[action](int(options.verbosity), options.interactive)
File "/Library/Python/2.5/site-packages/django/core/management.py", line 486, in syncdb
from django.db import connection, transaction, models, get_creation_module
File "/Library/Python/2.5/site-packages/django/db/__init__.py", line 11, in <module>
backend = __import__('django.db.backends.%s.base' % settings.DATABASE_ENGINE, {}, {}, [''])
File "/Library/Python/2.5/site-packages/django/db/backends/mysql/base.py", line 12, in <module>
raise ImproperlyConfigured, "Error loading MySQLdb module: %s" % e
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

So where as Python is installed by default under OS X, the mysql package is not.

How to install MySQLdb Package for Python.

First I went to the Mysql Downloads page and got a copy of the MySQLdb Package

Download the package, extract it, and open Terminal.app and change directories to the newly extracted package.

Here is where I ran into some fun, how do you install this? Well I found a website that suggested the Easy Installer…and that sounded nice, so I downloaded it first.

Download ez_setup.py, and run it; this will download and install the appropriate setuptools egg for your Python version.

There are many ways on install a package with the Easy Installer, I choose the directly from an unzip source folder, since that is what we have. And you can find more directions about the Easy Installer here

Now that you have Easy Installer, well installed, from terminal.app in the extracted MySQLdb Package MySQLdb folder run:
easy_install .

Got some errors, yeah we all did.

First, python will not be able to find your mysql installation (you did install mysql right?) If you didn’t get the .dmg file from the Mysql website, and install it through their very easy and very friendly installer, including a preference pane. The only problem with this it it puts itself in a non-standard folder, and that will cause a couple of issues.

Python will not be able to find mysql_config, or mysql for that matter because it is not in your path. To add it to your path add this line to the .profile file in your home directory.
PATH="${PATH}:/usr/local/mysql/bin"
Save that, and close terminal.app, and reopen it, This will cause the .profile to be re-read and your new addition to the path to be evaluated. Go ahead a try, now from the command line you should be able to call mysql_config, and any other mysql executable for that matter.

Try and run that easy_install command again, and yes you will still get errors.

Now you should get something like:
Running MySQL-python-1.2.2/setup.py -q bdist_egg --dist-dir /var/folders/At/At0AMaeKEAaYVW-LYfH6e++++TI/-Tmp-/easy_install-BUSsM_/MySQL-python-1.2.2/egg-dist-tmp-qFBtdU
In file included from /usr/local/mysql/include/mysql.h:43,
from _mysql.c:40:
/usr/include/sys/types.h:92: error: duplicate ‘unsigned’
/usr/include/sys/types.h:92: error: two or more data types in declaration specifiers
error: Setup script exited with error: command 'gcc' failed with exit status 1

Now to edit a couple of the files that came with the MySQLdb package. First _mysql.c
comment out the following lines:

#ifndef uint
#define uint unsigned int
#endif

Also edit the site.cfg file and change

threadsafe = True
to
threadsafe = False

Next, just one more Mysql Path fix run the following command:

sudo ln -s /usr/local/mysql/lib/ /usr/local/mysql/lib/mysql

Python will now be able to find the lib files where they should be. Ok that is it, run the easy_install . command and everything should go smoothly. As far a django is concerned, just continue to follow the walkthrough and you should do fine. I hope this helped.

Here are the websites that helped me, So if you run into any other issues these would be a good place to start: