Open Data - Benjamin Melançon

Resources and organizations of interest to Drutopia development

Benjamin Melançon — Wed, 13 Dec 2017 15:14:56 +0000

A biased list by Benjamin Melançon. Suggestions and edits welcome.

Drutopia is an initiative to build software for grassroots groups, in part funded and hosted by a platform cooperative.

Indieweb.org, home of the IndieWeb movement and IndieWebCamps to build the independent web. Focus on design first (good UX as the overriding concern in defining minimal formats to achieve a good experience for people interacting across independent sites), principles (and protocols) over project-centrism, and showing with prototypes rather than describing.
Redecentralize.org: An organization dedicated to decentralizing the Internet to make it resilient, private, and competitive/innovative.

Update: I also started an etherpad, editable by anyone, with a list of designer and developer communities to introduce to Drutopia.

In search of values

Benjamin Melançon — Thu, 30 Nov 2017 17:38:58 +0000

Here's a collection of value statements at diverse companies, as Agaric considers defining some more public values ourselves.

Nava PBC has a comprehensive set of well-thought-out values, entitled Nava's Core Values:

Be Active Stewards

The work we do impacts the lives of millions of people. The stakes are extremely high, so the standards we hold ourselves to must be even higher. In order to meet them, we must always listen closely to the communities we serve, and strive to develop a nuanced and insightful understanding of their experiences and needs. We must then continually advocate for these voices, both to ourselves and our partners. Only with great care and humility — a willingness to be wrong — can we build the tools that will be lasting, accessible, and truly helpful to the people who need them.

Pursue the Root Cause

We are here to build better foundations, not just better facades. The problems we solve are very complex, with broad social, cultural, and civic implications. This means we must always resist the quick fix in favor of pursuing the underlying cause. Getting at the root of a problem is not easy — it requires active listening, practiced empathy, and great tenacity – but it is absolutely necessary. Where we rebuild roots, we are able to effectively and efficiently remake systems from the ground up.

Think Long-term

We work to make basic government services effective, efficient, and responsive to the needs of millions of people. This is a continuous process — one that will outlast election cycles and resonate across lifetimes. We can’t and won’t think about our work in terms of the daily news cycle. Instead, we recognize that it benefits from, if not requires, taking a profoundly long-term perspective.

Build Together

True partnerships are truly collaborative. Our government partners bring decades of knowledge, experience, and policy insight to the table. We have our own set of specialities in technology and design. By bringing these skillsets together, our work becomes more impactful, the systems we build more resilient, and the tools we make more comprehensive. In order to achieve this collaborative spirit, we invite feedback early and often, always provide context for our choices, and share and receive knowledge with equal levels of enthusiasm. We are committed to always working in step with our partners in government.

Inclusion is Essential

Outcomes are unequivocally better when they incorporate a diverse array of insights and expertise. Recognizing that, we will always deliberately and thoughtfully pursue the widest range of viewpoints and life experiences possible. This helps us do the necessary work of recognizing and questioning our own assumptions. It also helps us find stronger and more holistic solutions to the problems we are striving to solve. The people we work with, and the range of experiences they bring to the table, should be directly reflective of the communities we serve. We cannot do our work well otherwise.

Progress Takes Work

Progress takes enduring commitment and extraordinary effort. It is composed of small, daily steps — not necessarily big leaps. We celebrate the difficulty of our daily work and aspire to its undertaking with enthusiasm. That often means tackling tasks that fall outside of the job description. It also means knowing how to thrive under pressure. We believe the future belongs to people who are willing to make change happen, and we are grateful to play our part.

Colab, a large worker-owned digital agency we admire, simply cribbed all their values from the United Nations Global Compact ten principles and the International Co-operative Alliance's definition of the seven cooperative principles.

The Ten Principles of the UN Global Compact

Human Rights

Businesses should support and respect the protection of internationally proclaimed human rights; and

make sure that they are not complicit in human rights abuses.

Labour

Businesses should uphold the freedom of association and the effective recognition of the right to collective bargaining;

the elimination of all forms of forced and compulsory labour;

the effective abolition of child labour; and

the elimination of discrimination in respect of employment and occupation.

Environment

Businesses should support a precautionary approach to environmental challenges;

undertake initiatives to promote greater environmental responsibility; and

encourage the development and diffusion of environmentally friendly technologies.

Anti-Corruption

Businesses should work against corruption in all its forms, including extortion and bribery.

The Co-operative Principles of The International Co-operative Alliance

Voluntary and Open Membership

Co-operatives are voluntary organisations, open to all persons able to use their services and willing to accept the responsibilities of membership, without gender, social, racial, political or religious discrimination.

Democratic Member Control

Co-operatives are democratic organisations controlled by their members, who actively participate in setting their policies and making decisions. Men and women serving as elected representatives are accountable to the membership. In primary co-operatives members have equal voting rights (one member, one vote) and co-operatives at other levels are also organised in a democratic manner.

Member Economic Participation

Members contribute equitably to, and democratically control, the capital of their co-operative. At least part of that capital is usually the common property of the co-operative. Members usually receive limited compensation, if any, on capital subscribed as a condition of membership. Members allocate surpluses for any or all of the following purposes: developing their co-operative, possibly by setting up reserves, part of which at least would be indivisible; benefiting members in proportion to their transactions with the co-operative; and supporting other activities approved by the membership.

Autonomy and Independence

Co-operatives are autonomous, self-help organisations controlled by their members. If they enter into agreements with other organisations, including governments, or raise capital from external sources, they do so on terms that ensure democratic control by their members and maintain their co-operative autonomy.

Education, Training and Information

Co-operatives provide education and training for their members, elected representatives, managers, and employees so they can contribute effectively to the development of their co-operatives. They inform the general public - particularly young people and opinion leaders - about the nature and benefits of co-operation.

Co-operation among Co-operatives

Co-operatives serve their members most effectively and strengthen the co-operative movement by working together through local, national, regional and international structures.

Concern for Community

Co-operatives work for the sustainable development of their communities through policies approved by their members.

Perhaps taking the position that living your values is more important than stating them, Palante Tech's about page simply describes what they do:

Palante Technology is a worker cooperative that provides tech consulting services to progressive nonprofit, social justice, activist and community organizations. Through this work we've developed a deep understanding of nonprofit tech needs. We have also been long involved with activism and organizing in non-technological capacities, including involvement with many of our client organizations. This breadth of experience allows Palante Tech to provide services that are tailored to meet the specialized needs of nonprofit community organizations.

Red Hat, likewise, does not have a single values statement but talks a great deal about "the open source way", with other value statements paired with actions taken sprinkled throughout their about page, such as:

ul>

Way down on Buffer's about page, beneath their philosophy, they offer "The 10 Buffer Values" which they say were influenced by Dale Carnegie (née Carnagey):

1. Choose Positivity and Happiness

You strive to approach things in a positive and optimistic way

You avoid criticizing or condemning team members or users

You avoid complaining

You let the other person save face, even if they are clearly wrong

You are deliberate about giving genuine appreciation

2. Default to Transparency

You take pride in opportunities to share our beliefs, failures, strengths and decisions

You use transparency as a tool to help others

You always state your thoughts immediately and with honesty

You share early in the decision process to avoid "big revelations"

3. Have a Focus on Self Improvement

You are conscious of your current level of productivity and happiness, and make continual changes to grow

You have a higher expectation of yourself than Buffer does of you

You regularly and deliberately do things that make you feel uncomfortable

You practice activities and develop habits that will improve your mind and body

4. Be a "no-ego" Doer

You don't attach your personal self to ideas

You let others have your best ideas

You approach new ideas thinking "what can we do right now?"

You are humble

You always ship code the moment it is better than what is live on our site - no matter what

5. Listen First, Then Listen More

You seek first to understand, then to be understood

You focus on listening rather than responding

You take the approach that everything is a hypothesis and you could be wrong

You are suggestive rather than instructive, replacing phrases such as 'certainly','undoubtedly', etc with 'perhaps', 'I think', 'my intuition right now'

6. Have a Bias Towards Clarity

You talk, code, design and write in a clear way instead of being clever

You over-communicate, repeating things more times than you would intuitively

You use more words to explain, even if it feels obvious already

You don't make assumptions, you instead ask that extra question to have the full picture

7. Make Time to Reflect

You deliberately find time for reflection, because that's where your life-changing adjustments come from

You have a calm approach to discussions and ponder points in your own time

You find time to jump out of the trenches into the higher level thinking that will move the needle

You understand the value of patience and treat it as a muscle which needs practice to grow

8. Live Smarter, Not Harder

You value waking up fresh over working that extra hour

You always aim to be fully engaged in an activity, or resting

You single task your way through the day

You are at the top of your game, as you focus on expanding capacity of your mental, physical, emotional and spiritual energy

You choose to be at the single place on Earth where you are the happiest, and most productive, and you are not afraid to find out where that is

9. Show Gratitude

You regularly stop and are grateful for your circumstances

You are grateful for the work co-workers do to push the company forward and help you move faster

You approach customer conversations with humility and the knowledge that it's a privilege to serve these people

You have gratitude for platforms, tools and open source that laid the foundation for the possibility of the company: "If I have seen further it is by standing on the shoulders of giants" - Isaac Newton

10. Do the Right Thing

You choose what's best for customers and the company in the long-term

You correct the mistake even when no one would notice

You strive to provide the best solution, even if that means foregoing profit

You get excited about opportunities to help others

Slack hides their core values on their career page, and doesn't elaborate on the one-word (plus emoji) per principle, beyond a brief introduction, which is a little depressing when you realize helping people "bring more of themselves to their work" is probably the best purpose you can have while being wildly successful under capitalism:

Our core values

These are some of the values we live by, as a company. We work by them, too: we’re building a platform and products we believe in — knowing there is real value to be gained from helping people, wherever they are, simplify whatever it is that they do and bring more of themselves to their work.

Empathy emoji Empathy
Courtesy emoji Courtesy
Thriving emoji Thriving
Craftsmanship emoji Craftsmanship
Playfulness emoji Playfulness
Solidarity emoji Solidarity

Agaric doesn't currently have a values, statement of purpose, or anything else publicly.

This is the closest we have on our present long-neglected web site:

Agaric helps people create and use powerful web sites. As a collective of skilled workers, we collaborate with you and free software communities to develop tools and build platforms that meet your needs.

I've floated "the most power possible to all people over their own lives" as a statement of purpose, but that's what i put down for everything i do— it's not unique to Agaric's particular ways of helping people gain control (let alone liberty and justice, to the extent we get to do that).

Agaric's inward-facing values, defined in a process led by Mauricio Dinarte and captured under the heading "Culture", are really solid and I think ought to be broadcast and lived more:

Culture

We prioritize health and family over work.

We work for the benefit of the group.

We encourage continuous learning.

We appreciate and welcome new ideas.

We give back to the communities we are part of.

We value long term relationships.

7 Cooperative Principles

Agaric is a cooperative and we work under these principles.

Voluntary and Open Membership

Democratic Member Control

Members' Economic Participation

Autonomy and Independence

Education, Training and Information

Cooperation among Cooperatives

Concern for Community

And as you can see we do our own summary of the seven cooperative principles; i think we decided NCBA had our favorite restatement of the seven cooperative principles but their link is broken.

Feedback on private, secure cloud storage / backup given to Least Authority

Benjamin Melançon — Thu, 21 Sep 2017 17:48:06 +0000

The convenience (can actually be used) and security balance is the real magic in this space. For example, i am at this moment fairly afraid that i've forgotten my SpiderOak passphrase and lost the backup piece of paper. On my ToDo list for months has been check if i can still access my data and then re-up the subscription. I own my failings, but i suspect other humans share them. Perhaps tiers of data security in one service could be an answer? One tier, the lowest, can be recovered with a password reset to e-mail, so the private key actually stored somewhere with you. Another tier, in the middle, could allow a shorter passphrase that is used frequently (and so actually remembered) with a backup key that is stored locally, encouraged to be printed, and even (for the many of us who are printer free) temporarily transmitted securely to your facilities and printed and sent by US mail to the user. The final tier is for the highest security and people who know they can hold secrets.

Time Tracking and Hours Reports in GitLab with GTT

Benjamin Melançon — Tue, 05 Sep 2017 10:52:25 +0000

GitLab has time tracking (caveat: Unfortunately this is the Enterprise Edition, not the fully Free Software Community Edition). See:

What it doesn't have is any evident way to see, in a reasonable overview, this time that's been tracked.

I got no responses asking this in #gitlab on IRC: Is there any UI for seeing tracked time by person? I can't find it (or any project overview of time tracked) and it's not documented if there is. As a fallback, is there a way to export this time estimate and tracked information? I'm on GitLab.com but we might go self-hosted at some point, so i'm interested on info about both. https://gitlab.com/gitlab-org/gitlab-ee/issues/1228 didn't have much.

Never did find the documentation for the API of getting tracked time. But i found an awesome tool that makes full use of that API:

https://github.com/kriskbx/gitlab-time-tracker

Installation

Installing that command line wonder was pretty easy.

Upgraded Node on my system with: https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions

Installed Yarn per this: https://yarnpkg.com/en/docs/install#linux

And then followed the installation instructions: https://github.com/kriskbx/gitlab-time-tracker#installation

Usage

gtt report "uc-davis/patient-hm-brain"

Project-specific configuration

In a .gtt.yml in the root of your project directory you can define the project that gtt should use so you don't have to type it each time, along with other options you want for the project. For instance:

project: nichq/nichq-community
showWithoutTimes: true
closed: true
query:
- issues
report:
- stats
- issues
- records

Here we have set the project (its path on GitLab.com), that we want it to show issues that do not have times (so we can see what we've estimated), to include closed issues (so we get the full time count), and to not include merge requests in queries or reports because we track all time on issues.

More usage examples

These examples require the project-specific configuration—at least the project being set—by a file like the one shown above.

View only a single sprint (milestone)

gtt report --milestone="Sprint Basinga"

Output a PDF

gtt report --milestone="Sprint Basinga" -o pdf -l basinga.pdf

Output CSV files

gtt report -o csv -l nichq-community-time-2017-12-08.csv

Different CSV files will be created for each section of the report, with the command above for example resulting in:

nichq-community-time-2017-12-08.issues.csv
nichq-community-time-2017-12-08.records.csv
nichq-community-time-2017-12-08.stats.csv

Get time worked for a particular user for a given time period

gtt report -u mlncn -f 2018-01-01 -t 2018-02-01

Searched words:

GitLab timetracking output report reporting estimated time time spent time worked

File:

gtt-output.png

Creating nicely formatted PDFs from Markdown files with Atom editor

Benjamin Melançon — Wed, 24 May 2017 14:38:53 +0000

Atom
Make sure Atom's core Markdown Preview package is enabled
Markdown to PDF for Atom
paste-table for Atom
LibreOffice Calc
(Optional) vim-mode-plus for Atom
(Optional) Make sure Atom's core autosave is enabled.

The current approach to having our proposals in version control is to generate PDFs from Markdown files. (This is much simpler to set up and use than our previous LaTeX approach, at the cost of losing some formatting control which we weren't using anyway.)

After installing Atom, install the above-referenced packages with:

apm install markdown-pdf paste-table vim-mode-plus

Or (if just starting with Atom) copy ben's Atom setup.

Searched words:

markdown preview markdown to PDF

Removing double spaces in usernames directly in the SQL

Benjamin Melançon — Fri, 05 May 2017 11:22:29 +0000

A client using the Bulk invite module (by Agaric's David Valdez) had invited hundreds of users with slightly malformed usernames— two spaces between first and last name when there should only be one.

Note that you need to do the change both in the users_field_data table and the users_field_revision table— and you need to clear caches when you're done to see the change on user edit pages, which is where it matters.

drush sqlc
UPDATE users_field_data SET name = REPLACE(name,'  ',' ');

Check for the response! For example, the first time i ran this it returned:

ERROR 1062 (23000): Duplicate entry 'John Doe-en' for key 'user__name'

This was because the same name was there twice (once with a single space and once with a double space) associated with different e-mail addresses (but almost certainly the same person). Rather than make the decision of which e-mail address should be used for this user, it was easiest to rename the existing username for the second address:

UPDATE users_field_data SET name = REPLACE(name,'John Doe','JohnDoe');

And then the original command could complete, like this:

mysql> UPDATE users_field_data SET name = REPLACE(name,'  ',' ');
Query OK, 243 rows affected (0.02 sec)
Rows matched: 497  Changed: 243  Warnings: 0

And then the users_field_revision table, in our case still with the extra step:

UPDATE users_field_revision SET name = REPLACE(name,'John Doe','JohnDoe');
Query OK, 1 row affected (0.01 sec)
Rows matched: 798  Changed: 1  Warnings: 0
mysql> UPDATE users_field_revision SET name = REPLACE(name,'  ',' ');
Query OK, 250 rows affected (0.01 sec)
Rows matched: 798  Changed: 250  Warnings: 0

Then exit MySQL/MariaDB/whathaveyou and clear caches:

../vendor/bin/drush cr

Searched words:

SQL search for double space MySQL find and replace doublespace in database field

Send gitlab.com commit message to IRC

Benjamin Melançon — Fri, 10 Mar 2017 03:42:54 +0000

There is sadly no default way to do this directly from GitLab.com— but there's a pretty simple way, with Bishop:

https://github.com/ta/bishop

Install Heroku locally. These instructions from https://devcenter.heroku.com/articles/getting-started-with-ruby#set-up worked great for me:

sudo add-apt-repository "deb https://cli-assets.heroku.com/branches/stable/apt ./"
curl -L https://cli-assets.heroku.com/apt/release.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install heroku

Then:

heroku apps:create bishop-example --stack cedar

Results in:

Creating ⬢ bishop-example... done, stack is cedar-14
https://bishop-example.herokuapp.com/ | https://git.heroku.com/bishop-example.git

heroku config:add BISHOP_API_KEY=dc83Pzn2foy4039 BISHOP_SERVER=irc.freenode.net BISHOP_CHANNELS=#example

Setting BISHOP_API_KEY, BISHOP_SERVER, BISHOP_CHANNELS and restarting ⬢ bishop-example... done, v3

BISHOP_API_KEY: dc83Pzn2foy4039
BISHOP_CHANNELS: #example
BISHOP_SERVER: irc.freenode.net

git push heroku master

(Note: The heroku app adds heroku as a remote repository all on its own.)

And, this is critical:

heroku config:add BISHOP_GITLAB_HOOK_CHANNELS=#example

We hacked it to rename the bot to bishop-example
2
3 # Changes to be committed:
4 # modified: lib/bishop/bot.rb

git push heroku

Now, to a GitLab repository, under settings/integrations, add a webhook:

https://bishop-example.herokuapp.com/hooks/gitlab/dc83Pzn2foy4039

Date field date-only migration from Drupal 7 to Drupal 8

Benjamin Melançon — Mon, 06 Feb 2017 18:51:04 +0000

After we did the upgrade of a Drupal 7 site to Drupal 8, a date field which had been taking only the date portion, and no time portion, was now configured to take both the date and the time. Moreover, the migration wasn't working.

We changed the configuration from datetime to date

Here's the relevant part of the diff for config/default/field.storage.node.field_pub_date.yml

 field_name: field_pub_date
 entity_type: node
 type: datetime
 settings:
-  datetime_type: datetime
+  datetime_type: date
 module: datetime
 locked: false
 cardinality: 1

Note that changing this configuration requires a re-install of the site— storage settings can't be changed in an operating site.

The date was actually imported but, as shown by a "SELECT field_pub_date_value, entity_id FROM node__field_pub_date;" query, in the format "2016-10-10 00:00:00". That caused it to not display, and when editing, to show the field as blank. Saving the node after filling out the blanked field showed that it was saved as "2016-10-10" in normal operation.

In config/default/migrate_plus.migration.upgrade_d7_node_article.yml changing the line the Drupal upgrade had left us, "field_pub_date: field_pub_date" under the process directive, to the following instead, fixed the date migration:

process:
  field_pub_date:
    plugin: iterator
    source: field_pub_date
    process:
      value:
        plugin: substr
        source: value
        start: 0
        length: 10

Searched words:

drupal 8 date migrate drupal 8 migrate datetime field not migrated drupal 7 to drupal 8 "date" migration The input value must be a string.

Using JavaScript in Behat tests (manual steps for installing, running)

Benjamin Melançon — Mon, 14 Nov 2016 15:25:40 +0000

In virtual machine:

cd ~

# Go to http://phantomjs.org/download.html to see the latest available.
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

tar -xjf phantomjs-2.1.1-linux-x86_64.tar.bz2

# phantomjs --webdriver=4444

/home/vagrant/phantomjs-2.1.1-linux-x86_64/bin/phantomjs --webdriver=4444

Now you can test @javascript tagged Behat tests.

Until the above is automated, exclude JavaScript-requiring tests by default.

In behat.yml:

default:
  gherkin:
    filters:
      tags: ~@javascript

Run with:

behat --tags=javascript

References:

http://docs.behat.org/en/v3.0/guides/6.profiles.html

http://jaffamonkey.com/using-phantomjs-with-behat/

Searched words:

behat drupal extension contextual links edit debian extract tar.bz2 behat.yml exclude by default

Easily add content with update hooks: use default content module exports to create content that needs to be in sync with configuration

Benjamin Melançon — Mon, 19 Sep 2016 19:17:55 +0000

An approach for keeping content that is added as part of development synchronized with test and production environments is to use the Default Content module to export the content. It's built for the content to be exported to an installation profile's 'content' folder, and then the module, if enabled, automatically brings the content in when the site is installed. Presuming that reinstalling the site would be a drastic step for getting one block's content (aside: we do find it useful to use installation profiles with default content for automated testing and initial development), it's also possible to import the content one item at a time, such as in an update hook, with the below code in your example.install or example.profile:

Updated to support taxonomy terms import!

<?php
/**
 * Import a piece of content exported by default content module.
 */
function example_import_default_content($path_to_content_json) {
  list($entity_type_id, $filename) = explode('/', $path_to_content_json);
  $p = drupal_get_path('profile', 'guts');
  $encoded_content = file_get_contents($p . '/content/' . $path_to_content_json);
  $serializer = \Drupal::service('serializer');
  $content = $serializer->decode($encoded_content, 'hal_json');
  global $base_url;
  $url = $base_url . base_path();
  $content['_links']['type']['href'] = str_replace('http://drupal.org/', $url, $content['_links']['type']['href']);
  $contents = $serializer->encode($content, 'hal_json');
  // The path for the taxonomy term class is different.
  if ($entity_type_id == 'taxonomy_term') {
    $class = 'Drupal\taxonomy\Entity\Term';
  } else {
    $class = 'Drupal\\' . $entity_type_id . '\Entity\\' . str_replace(' ', '', ucwords(str_replace('_', ' ', $entity_type_id)));
  }
  $entity = $serializer->deserialize($contents, $class, 'hal_json', array('request_method' => 'POST'));
  $entity->enforceIsNew(TRUE);
  $entity->save();
}
?>

Export a custom block with an ID of 8:

drush dcer block_content 8

And used in your example.install file:

<?php
/**
 * Add the footer block content.
 *
 * Implements hook_update_N().
 */
function example_update_8001() {
  example_import_default_content('block_content/136efd63-021e-42ea-8202-8b97305cc07f.json');
}
?>

Searched words:

programmatically make nodes, taxonomy terms, menu items, other content