NYC Workshop Notes, May 2014

From wiki.measurementlab.net
Jump to: navigation, search


Measurement Interests and Prospective Methodologies

Measurement Platforms and Initiatives

Infrastructure and Testing Consideration

Initial Plans and Subjects for Discussion

Thursday Morning Session Notes

  • No one should be able to infer political intent because of the performance of censorship measurement tests:
    • Should be similar to motivations regarding network neutrality; or,
    • Why don’t conservative Christians organizations measure censorship?
  • Clear need across projects and research applications for longitudinal standards and baseline, to allow for framing in terms of Internet health or differential access changes;
  • Should take the politics out of measurement, which occurs especially when measurements only target the Middle East and around particular events;
  • Units of reporting or observation matter since censorship occurs at lower levels than a country-basis, schools and organization censor at the lowest level;
  • Remaining questions across projects about the level of reporting: raw data versus metadata versus abstracted information;
  • Destigmatization may be partially accomplishable through labelling and language, words such as health and interference (latter seen with the naming of OONI):
  • Dry language is helpful toward this purpose, but not as motivating or funded as ‘sexier’ branding of censorship,
  • Speaking in terms of a gradient versus a binary or dichotomous situation as well;

Friday Morning Session

  • On the use of rPis, presentation matters, BISmark trials had resistance based on the use of a grey box -- problematic since people expect a network device to look a certain way,
    • This may not be an issue since the incentive is different (free of expression appeal versus a free device);
  • Requests for dashboard to demonstrate the impact of participate in measurement efforts, could be linkable with measurement IDs without deanonymization,
  • Projects may need fresh documentation review by others not versed in the project’s development,
  • Map expectations ahead of time for testing and development of projects,
  • Need for modular infrastructure for helpers, collectors -- however, changes in infrastructure have already burned projects,
    • Including helper stability and compatibility;
  • Canonical DNS answers and mapping will help with assessing measurement validity,
  • Common interests in sources of lists to measure and directly comparable data formats,
  • Could measurement efforts be bundled together -- if one person is interested in one measurement initiative, they may be willing to perform others;

Milestones

General Themes Across Projects:

  • Tools that produces useful data,
  • Publishing of the raw data within appropriate boundaries,
  • Analysis and interpretation of the data,
  • Education, promotion of the interpretations of the data.

OONI

Wishlist

  • Share test helpers/collectors.
  • Write docs to improve this reuse, share and do not recreate.

Objectives

  • A tool to generate useful data.
  • People will look at the data and interpret it.
  • Based on the interpretation (“translation”) of such data, the best decisions are made.
  • There is more transparency on the economics of surveillance/censorship tech.
  • What changes in the world?
    • Reduce the amount of over-censoring.
    • Shift in people’s interpretation of censorship.
    • Measuring interference goes mainstream.
  • Make this data more understandable

Development Calendar

May

  • Add crontab to deb that regularly tries to upload stored test data
  • Who (user)(uid) runs cron tasks?
  • Decide when to run test and submit data
  • Add mechanism for feedback to user to “see if it’s working”

June

  • Build stable .deb for Debian Wheezy
  • How where to port .debs for users to install
  • Auto-building OONI packages for autobuild .deb for target (wheezy)
  • Document install process (and test/refine documentation)
  • Set up and test collectors on M-Lab slice (test rel.)
  • Build and/or retool monitoring infrastructure
  • Magic to (pipeline) hook collector into M-Lab data pipeline
  • Format conversations for YAMLoo to BigQuery

July

  • Q/A check the .deb install pathway for Ooni-client
  • Resolve M-Lab bouncer overhaul questions
  • Run through full test pathway on 2+ M-Lab nodes, with 2+ clients (manual, test related)

August

  • Integrate all backend code into ooni repo, which will be used to build slice package
  • Integration tests for successful propagation of data through entire collection pipeline (integration related)

September

  • Q/A test slice package
  • Monitor initial client deployments for success (fail to public, fail to test, fail to parse, etc) (integration related)

Wish List

  • Find OONI on other milestones and answers for integration questions
  • ICLab reuse/duplicate OONI code
  • Share test helpers/collectors
  • Write docs to improve reuse -- encourage other tests to share do not recreate tests

Development Discussions

Thursday

  • Primary Questions:
    • Can we have mlab-ns monitor helpers, collectors, and bouncers as separate helpers – or – can we return multiple fields?
    • Can we have mlab-ns return .onions, such as in the FQDN field?
    • Can we do the two above with minimal modifications to mlab-ns?
      • If we modify mlab-ns, can we modify it in a way that's most likely to be most helpful to future experiments?
      • How little can we modify OONI to implement the above?
    • Does mlab-ns need to return a custom FQDN?
  • Approach One:
    • A new bouncer makes a broad query for all OONI stuff, then determines how to answer probe queries,
    • Ooni: Open a ticket for making this custom bouncer -or- modify the collector to return information about the other helpers in the process.
    • M-Lab: Open ticket for M-Lab, to alter the query to return JSON with all services (or all Ooni services),
      • Add tickets to mlab-ooni support for the mlab-ns tweaks.

Friday

  • Milestones:
    • Wheezy .deb,
      • user in debian policy (where is $HOME),
      • Q/A check deb installs
      • determine deb repository;
    • Measurement Lab Integration
      • M-Lab monitoring integration,
      • mlab-ns integration,
      • deploy collectors/helpers on slices,
      • integration test propagation of data
    • Run "beta test" which allows us to do a sanity check on the data / smoke test,
    • Improve integration tests to verify data is collected,
    • End-to-end integration tests are an M-Lab criterion,
    • Add user feedback mechanism,
    • Success involves incorporating the data from oonib into the M-Lab db -- magic pipeline and format .yml within BigQuery,
    • Crontab in debian policy,
    • Determine when do we run the tests -- e.g. upon reboot and when in cron,
    • Ensure that if network is down, it does the right thing, where the data allows us to distinguish ifconfig eth0 down from interference,
    • Documentation on how to edit ``/etc/apt/sources.list`` and install the wheezy .deb,
    • Monitor initial deployments,
    • Integrate backend code into ooni-support package,
    • Autobuild debs (talk to weasel),
      • debs are already made for dependencies not in wheezy, then integrate them into weasel's autobuild;

Further Notes

ICLab

Wishlist

  • Crontab (connect with Sam @ BISmark and OONI),
  • Integrate with OONI now that “we” are using python,
  • Shared scheduling,
  • Python on Android,
  • Visualization and dashboard,
  • Test helpers (OONI),
  • Common data pipeline (ICLab), data format and transport,

Objectives

  • Bridging between technical and advocacy/policy/impact:
    • (Short term) Having measurements running on certain devices in X countries,
  • Reproducible standard method,
  • Agility of measurement:
    • When, where, what, how,
  • Stable base-test,
  • Make information available for advocacy:
    • Multi-resolution,
  • No more one-off, paper projects,
  • Extensibility,
  • Archiving Data.


Development Calendar

May

  • Get WA + Peny Fns Working
  • Load Sathyas’ code on rPi
  • Get web UI + measurements on same rPi
  • Perform Userbase study for deployment

June

  • Determine what we can steal from OONI now that (we are using) Python
  • Get rPi sending data back to data server for analysis
  • Get rPi’s communicating with controller receiving experiments/comments
  • Get tests that the lab is already running running daily on Pis (synced HTTP tests, DNS, traceroute)

July

  • create a scheduler that makes cronjobs
  • automate informed consent procedure (risk: levels, link consent and risk level to the rPi)
  • Get experiments running for Indonesian election

August

  • Look into test helper servers/overlap with OONI
  • Figure out and implement a sandboxing technique
  • Have a debian package for background tests
  • Multi Platform support Linux (Mac?)

October

  • Multi-platform support: Android, iOS
  • Web interface for researchers

December

  • Multi-platform support: Windows

May 2015

  • Integrate the differentiation detection App into the platform

CitizenLab

Wishlist

  • Wants use documentation from OONI,
  • Willing to audit usability from a user’s perspective.

Least Authority

Wishlist

  • Find partners to run code bundle probes together
  • Common analysis
  • Visualization and dashboard
  • Pipeline and data format
  • Python collaboration
  • Common data pipeline (ICLab), data format and transport,

Encore

Objectives

  • Goal: thousands of clients per day performing measurements in dozens of interesting countries.

Wishlist

  • Collaboration on lists of things to measure (ICLab, etc.)
  • Data Storage (M-Lab, etc.)
  • Visualization and dashboard,

Development Calendar

May

  • Talk to Zooko about service monitoring deployment
  • Explore issues blocking encore from doing service monitoring

June

  • Develop per-country measurement lists (do we know they are safe?)

July

  • Build more crawlers (and figure out whether we need them)
  • UI for showing sites we measure (targets)
  • Do a PR push
  • Write a blog post

September

  • Send encore data to M-Lab for store

Deployment Strategies

  • Recruit more professors,
  • Get Slashdotted,
  • Brand as service monitoring,
  • Buy some ads.

My Speed Test

Wish List

  • Python on Android
  • Sharing test cases with OONI and ICLab
  • Common data pipeline (with ICLab)
  • Common scheduling language (with ICLab)

Development Calendar

May

  • Push newest fixed version (minus beta features)
  • PR pushes

July

  • Improve DNS consistency test by using reverse lookup

August

  • Figure out (Kivy) python on Android
  • Collect DNS resolver used

September

  • Figure out transport for rsync on Android
  • Figure out what tests exist

October

  • Common scheduler w/ Raspberry Pi
  • Integrate MST to a common data pipeline with ICL

November

  • Implement HTTP header manipulation test

December

  • Notification for real time monitoring


Satellite

Wish List

  • Storage on M-Lab
  • Common data format
  • Correct IPS for DNS queries to check for spoofing v. dual homing/anycasting

Development Calendar

June

  • Analysis of what a correct DNS response is,
  • Find keyword dictionary for keyword blocking test.

July

  • Decide correct log/data formats especially for high bandwidth tests.

August

  • Determine most accessible data storage location (S3 v. M-Lab v …).

September

  • HTTP keyword blocking test,
  • Analysis of data to make comparable to existing reports.

November

  • Automate and package data collection program,
  • Create reference data analysis program.

December

  • Release data and Code.

May 2015

  • Get people to contribute tests and use data.
  • find partners to run code

Measurement Lab

Wishlist

  • Encore data on M-Lab and host crawlers
  • Dashboard for results over time
  • Share code with our work on browser extension
  • Satellite data on M-Lab
  • ICLab /OON/ helper integration help

Objectives

  • When the platform is well-documented and built for others to use for censorship research.
  • Hosting compelling experiments that produce data that anyone can use.
  • Produces verifiable & repeatable/reproducible data.
  • Live data
  • Catalyze collective action
  • Be the host for resources that provide definitive censorship “status”
  • Bundle resources to make sum > parts
  • Global dashboard
  • enabling civil-society to act on the data generated by the platform
  • exposing censorship ⇒ reduction in global censorship and in acceptance of censorship

Wishlists

  • define audience(s)
  • documentation
  • examples
  • what is possible?
  • API for data access --->
  • API for experiments ---> appropriate to audience
  • stabilization
  • new experiments/automation of deployment
  • adoption (interaction w/researchers, experiment writers, gov’t, CSGs, users) = awareness
  • of stance on neutrality
  • competitiveness among researchers
  • end-user visualization
  • buy-in
  • don’t overly reimplement
  • from many people (researchers, commercial service providers, users)

Other Documentation