I was reading through the Communication of the ACM today.  I like skimming the articles.  Sometimes I learn something.  More often I see trends in thought, which then makes me think about where that will lead "us" in time...

Communications of the ACM.  March 2011 Volume 54 No. 3

In the letters to the editor Basudeb Gupta espouses that all software people (he used engineers) should be classified as engineers and be required to be certified, retrained, and controlled by licensing, much as Medical Doctors and Engineers are.  It's an argument I've heard off and on through my career, which isn't as long as his, but is sufficiently long to know why he's espousing his viewpoint.  Given the footnotes to articles published in the Communications of the ACM and their dates, this is a long standing argument for him.

I don't disagree on some levels, but to keep people from practicing without licensing, I disagree.  I think it's the equivalent of censuring software. You also run the risk of blotting out, ignoring, and pushing to the side valid approaches because those who control the process disagree with the value or validity. [look at difference in the philosophy of approach between Eastern and Western medicine - then look at the history of how and where they are practiced and/or controlled].

World Wide Web

Several years ago my daughter came home from a friend's house totally incensed.  She wanted us to sign a petition, she wanted us to broadcast this horrid event to our friends and get them to sign a petition, she wanted to have "something done" about this event... She and her friend had been told about a blog post where the person had written an article about an art show in Argentina.  Supposedly one of the artists had chained a dog in the corner of the exhibit and starved it to death... as a form of art.

So, her father and I spent some time researching "this event".  The blog post was written by an individual.  They weren't associated with a newspaper, they weren't associated with the museum in question.  They were "just an individual".  We looked further. We found a news article written for a reputable news agency (well as reputable as any of them get).  They hadn't gotten hold of the artist, but they had gotten hold of the museum director.  His story was... the artist had picked the dog up off the street - it was starving when he found it.  He did tie it in the museum - to raise awareness about the ignored populace of dogs in the city - that were starving.  The dog had been fed and watered, it gained weight while "on exhibit".  It had escaped one night, and so "disappeared" from display.

Who would you believe?

Logos forum

Logos converts printed material into researchable and cross-referenced electronic books.  They concentrate in the religious/historical sector. Someone made a request for a specific book to be added to the "library of research material" supported by Logos.  It is an English translation of a Syriac text.  My husband spent most of a morning pulling together a carefully constructed comment on the quality of the particular book's translation specifically in relation to *research* to be used by an unknown number of people with likely a very low level of vertical knowledge of the field. [Specifically Syriac language translation].  In short, his detail review of the material exposed some significant failures to conform with accepted interpretation of portions of the text.

How would they know whether the source was sound?

Communications of the ACM.  March 2011 Volume 54 No. 3

On pages 12 and 13 in the blog@ACM section there were two articles that dove tail into this discussion...

Mark Guzdial : What do Scientists and Engineers need to Know about Computer Science?

Chris Scaffidi, Mary Shaw, and rad Myers have estimated that, by 2012, there will be about three million professional software developers in the U.S, but there will also be about 13 million end-user programmers - people who program as part of the their work, but do not primarily develop software.
...
When they see "try-catch" in a piece of code that they're trying to understand, they don't know how to look up "exception handling", and they can easily spend hours reading about Java exception handling when they are actually working in JavaScript.

How close does this relate to my problem?

Greg Linden: Research in the Wild: Making Research Work in Industry

To take one example in search, without researchers who know the latest work, it would be hard for a company to build the thousands of classifiers that ferret out subtleties of query intent, document meaning, and spamminess, all of which is needed for a high-quality search experience.  Information retrieval is a field that benefits from a long history of past work, and researchers often are the ones that know the history and how to stand on giants' shoulders.

Who is an expert, what are their bias's?

Wikipedia

As I understand it... Wikipedia uses a structured review process to validate article content while allowing virtually anyone to say anything about anything. They depend upon balanced review by interested parties to argue with the content and revise it towards some level of quality result.

Is this a valid approach, are the results valid?

Ebay

Ebay instigated a feedback system and a arbitration process to allow the populace to police themselves.  If people generally have a good reaction to working with a specific person and they feel like telling people about it...

Are they trust-worthy...enough?  What are my guarantees?

Mankind has traditionally assigned "rank" to a person through birth or personal achievement which resulted in a visible standing. [material assets, wealth or titles]  Along with that comes some level of "trust" factor on the validity of their input about any specific topic.  But those too can be flawed.  Titles can be given for the wrong reason.  Assets can be acquired without the attendant skill to amass them [inherited wealth].

With the World Wide Web's low bar for entry... We have a world full of monkeys typing at a typewriter, and occasionally someone publishes a masterpiece.  But how do we find it and how do we know it's a masterpiece?

The question comes down to "How do we know whether to trust the content of what we are reading?"

Where does this lead?  While the search engines rank results in searches, they can't and don't apply any secondary validity rank on what is supplied. But then, should they?  "Who died and made you king?"

Information validity or source integrity, not only in the "wild woolly world of the web", but internal to organizations is becoming of huge concern with the exponential growth of electronic noise available to us. How we capture it, how we clean it, how we classify it, how we search it, and how we connect it to other results will continue to be vital to finding those nuggets of value and identifying those who produce more nuggets vs noise.

Reputation.  In a small community, reputation determines the value of their input. Generally people care about their reputation, they nurture it.  When that is migrated into a large community, it gets diluted in strength for the majority of the people, and concentrated into the few who are visible.

Somehow, we need to be able to assign a "pecking order" to the articles/data/information that comes our way, based on the perceived quality of the source.