Commit 7c492320 authored by cazenave's avatar cazenave
Browse files

Initial commit

parents
/module
/target
.git
.gitignore
.dockerignore
Dockerfile
.idea
*.iml
LICENSE
README.md
.travis.yml
# Compiled class file
*.class
# Log file
*.log
# BlueJ files
*.ctxt
# Mobile Tools for Java (J2ME)
.mtj.tmp/
# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
# build folders
target/
module/MOD-INF/classes/
module/MOD-INF/lib/
!project-repository/**/*.jar
# IntelliJ IDEA
.idea
*.iml
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- Possibility to specify custom storage in frontend
- History of created metadata in new dialog and stored using overlay model
per project
### Changed
- Saving project when overlay model updated (history)
- Localizable audit table headers
### Fixed
- Virtuoso storage can use HTTP Basic Auth
## [1.4.0]
### Changed
- Verified compatibility with FAIR Data Point 1.4.0
## [1.3.0]
### Added
- Audit log (per-project) with possibility to browse, filter, and clear using
a simple dialog in client-side
### Changed
- Prepared getting and passing SHACL specs from FDP to OpenRefine frontend
- Updated for compatibility with newer FAIR Data Point 1.3.0
### Fixed
- Persisting information about stored metadata
## [1.2.0]
### Added
- Possibility to restrict filesize and naming (using patterns) for storage
- Autofill file-related fields with content type and bytesize of stored file using
"Store FAIR data" button
### Changed
- Updated for compatibility with FAIR Data Point v1.2
- Removed fairmetadata4j
### Fixed
- Clear configuration before reloading it
## [1.1.0]
### Added
- Remembering the last used catalog and dataset per repository (FDP) persistently for the
OpenRefine project
- Display version and build info of the connected FAIR Data Point if present
- Use `instanceUrl` of FAIR Data Point if present to allow different domains for FDP and for
its repository metadata
- Information in About dialog derived from POM and project.properties files
- Support for different OpenRefine version in Docker image using [ARG](https://docs.docker.com/engine/reference/builder/#arg)
- Service tasks endpoint for reloading configuration files (extensible for the future), currently
intended for testing purposes only (end-to-end tests)
### Changed
- Updated to [OpenRefine 3.3](https://github.com/OpenRefine/OpenRefine/releases/tag/3.3)
### Fixed
- File storing using FTP (file extensions and corrupted binary files)
- Storing metadata in FAIR Data Point (in some cases error occurred)
## [1.0.0]
Initial version based on reproducing functionality of deprecated [FAIRifier](https://github.com/FAIRDataTeam/FAIRifier).
### Added
- Create metadata dialog that handles the connection with a FAIR Data Point and allows
metadata selection and creation with a possibility to predefine connections in the
configuration
- Metadata forms to create metadata in the selected FAIR Data Point, with typehints and
ability to predefine value in the configuration
- Store data dialog that allows to export project in the selected format (using OpenRefine's
exported) and send it to the selected storage from those which are configured - currently
supports: FTP, Virtuoso, and Triple Stores with HTTP API
- About dialog with basic information about the extension and its compatibility
- Report a bug link to create a GitHub issue easily
[Unreleased]: /../../compare/v1.4.0...develop
[1.0.0]: /../../tree/v1.0.0
[1.1.0]: /../../tree/v1.1.0
[1.2.0]: /../../tree/v1.2.0
[1.3.0]: /../../tree/v1.3.0
[1.4.0]: /../../tree/v1.4.0
# Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to make participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of experience,
nationality, personal appearance, race, religion, or sexual identity and
orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team. All complaints will be reviewed and
investigated and will result in a response that is deemed necessary and appropriate
to the circumstances. The project team is obligated to maintain confidentiality
with regard to the reporter of an incident. Further details of specific enforcement
policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at [http://contributor-covenant.org/version/1/4][version]
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
# Contributing
When contributing to this repository, please first discuss the change you wish to make via issue,
email, or any other method with the owners of this repository before making a change.
Please note we have a code of conduct, please follow it in all your interactions with the project.
## Pull Request Process
1. Ensure any unnecessary install or build dependencies and other files are removed before the end of
the layer when doing a build.
2. Explain the changes and update the README.md file and other documentation if necessary.
3. Be ready to communicate about the Pull Request and make changes if required by reviewers.
4. The Pull Request may be merged once it passes the review and automatic checks.
## Gitflow Workflow
We use the standard [Gitflow Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow):
* __master__ branch is used only for releases (and eventually hotfixes), this branch is also protected on
GitHub (pull requests with review and all checks must pass)
* __develop__ branch is used for development and as a base for following development branches of features,
support stuff, and as a base for releases
* __feature/*__ (base develop, squash-merged back to develop when done)
* __support/*__ (like the feature but semantically different, not the feature but some chore, e.g., cleanup or
update of Dockerfile)
* __release/*__ (base develop, merged to master and develop when ready for release+tag)
* __hotfix/*__ (base master, merged to master and develop)
Please note, that for tasks from [our Jira](https://dtl-fair.atlassian.net/projects/FOR/issues), we use
such as `[FOR-XX]` identifying the project and task number.
## Release Management
For the release management we use (aside from the [Gitflow Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)):
* [Semantic versioning](https://semver.org)
* Release Candidates - X.Y.Z-rc.N should be created if don’t expect any problems (in that case use alpha or
beta), and make a walkthrough to verify its functionality according to the manuals finally - it also verifies that the documentation is up to date with the new version.
* [CHANGELOG.md](https://keepachangelog.com/en/1.0.0/ )
* GitHub releases and tags - make the release using GitHub (or hub extension), Travis CI will automatically upload
ZIP and TGZ distribution files there - better verify.
* Docker Hub image - in case of release, Docker image with the same tag will be created automatically.
Also, never forget to update the joint [FAIR Data Point documentation](https://github.com/FAIRDataTeam/FAIRDataPoint-Docs)!
## New OpenRefine Versions
During the life-cycle of this extension, the OpenRefine will evolve in parallel. To handle this correctly, in case
of new OpenRefine version, we use this process:
1. Start new support branch when necessary changes are made to use the newer version of OpenRefine. Don’t forget to
update `project-repository`, assembly, documentation, and `Dockerfile`. Then merge to develop.
2. Create a new release with higher version according to OpenRefine version and semantic versioning (major breaks
compatibility, minor with new features, patch just hotfixes).
3. Filter out unfinished work that is no longer needed because of new OpenRefine (if any).
4. Update all other branches.
5. From now on, the newer version of OpenRefine is supported. For future releases, you should indicate if it works also
with the previous or not. In case of need, a new branch __legacy/OpenRefine-X.Y__ can be created for hotfixes.
_NOTE_: This still needs to be verified by practice!
# Build image
FROM maven:3.6-jdk-8-slim as builder
# You can specify OpenRefine version by: --build-arg OPENREFINE_VERSION=X.Y
# Possible versions: 3.3 (supported, default), 3.2
ARG OPENREFINE_VERSION=3.3
WORKDIR /usr/src/app/
# Add necessary project parts (exclude in .dockerignore)
COPY . .
# Compile and create package
RUN mvn clean package
# Prepare OpenRefine and metadata extension
RUN curl -sSL https://github.com/OpenRefine/OpenRefine/releases/download/$OPENREFINE_VERSION/openrefine-linux-$OPENREFINE_VERSION.tar.gz | tar xz
RUN mv openrefine-$OPENREFINE_VERSION openrefine
RUN tar xzf target/metadata-OpenRefine-3.3.tgz --directory openrefine/webapp/extensions
# ===================================================================
# Main image
FROM openjdk:8-jre-slim
# Dependencies for running OpenRefine
RUN apt-get -qq update \
&& apt-get install -qq -y --no-install-recommends wget=1.20.* \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Copy prepared OpenRefine with extension
COPY --from=builder /usr/src/app/openrefine /app
COPY src/main/resources/module/config/settings.yaml /app/webapp/extensions/metadata/module/config/settings.yaml
# Prepare workspace volume
VOLUME /data
WORKDIR /data
# Expose OpenRefine's port and run it
EXPOSE 3333
ENTRYPOINT ["/app/refine"]
CMD ["-i", "0.0.0.0", "-d", "/data"]
MIT License
Copyright (c) 2019 FAIR Data Team
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# OpenRefine-metadata-extension
[![OpenRefine metadata CI](https://github.com/FAIRDataTeam/OpenRefine-metadata-extension/workflows/OpenRefine%20metadata%20CI/badge.svg?branch=master)](https://github.com/FAIRDataTeam/OpenRefine-metadata-extension/actions)
[![GitHub All Releases](https://img.shields.io/github/downloads/FAIRDataTeam/OpenRefine-metadata-extension/total)](https://github.com/FAIRDataTeam/OpenRefine-metadata-extension/releases)
[![Docker Pulls](https://img.shields.io/docker/pulls/fairdata/openrefine-metadata-extension)](https://hub.docker.com/r/fairdata/openrefine-metadata-extension)
[![License](https://img.shields.io/github/license/FAIRDataTeam/OpenRefine-metadata-extension)](LICENSE)
[![Documentation](https://readthedocs.org/projects/fairdatapoint/badge/?version=latest)](https://fairdatapoint.readthedocs.io/en/latest/openrefine/usage.html)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/aca649b193144fb68428ba3039a49ad5)](https://www.codacy.com/manual/MarekSuchanek/OpenRefine-metadata-extension?utm_source=github.com&utm_medium=referral&utm_content=FAIRDataTeam/OpenRefine-metadata-extension&utm_campaign=Badge_Grade)
[OpenRefine](http://openrefine.org) extension to support **FAIR Metadata** with use of [FAIR Data Point](https://github.com/FAIRDataTeam/FAIRDataPoint)
## Installation
Distribution ZIP/TGZ files are located at [releases](https://github.com/FAIRDataTeam/OpenRefine-metadata-extension/releases) page.
Just download it and unzip/untar into the `extensions` folder in your OpenRefine instance, for more information visit [OpenRefine - Installing Extensions](https://github.com/OpenRefine/OpenRefine/wiki/Installing-Extensions).
```console
$ unzip metadata-X.Y.Z-OpenRefine-3.3.zip path/to/openrefine-3.3/webapp/extensions
# or
$ tar xzvf metadata-X.Y.Z-OpenRefine-3.3.tgz -C path/to/openrefine-3.3/webapp/extensions
```
### Via Docker
We publish [Docker images](https://hub.docker.com/r/fairdata/openrefine-metadata-extension) that contains OpenRefine 3.3 together with the **metadata** extension. Only requirement is having Docker service running:
```console
$ docker run -p 3333:3333 fairdata/openrefine-metadata-extension
```
Then just open [localhost:3333](http://localhost:3333) in your favorite web browser. Optionally you can change the port binding or run it "detached". Visit Docker `run` [documentation](https://docs.docker.com/engine/reference/run/) for more information.
To persist data and eventually be able to share them across multiple instances of OpenRefine, you need to mount `/data` directory, for example:
```console
$ docker run -p 3333:3333 -v /home/me/openrefine-data:/data:z fairdata/openrefine-metadata-extension
```
To add other extensions (e.g. [RDF extension](https://github.com/stkenny/grefine-rdf-extension)), you can just put them into the mounted folder according to the official [documentation](https://github.com/OpenRefine/OpenRefine/wiki/Installing-Extensions). Always also check installation instruction of the desired extension. For the previous example, you should place your extensions to the directory `/home/me/openrefine-data/extensions`.
### Configuration
Configuration files are located in `extensions/metadata/module/config` folder and examples are provided. In case of Dockerized instance, you need to mount this folder with your configuration files as volume similarly to `data` folder.
For more information, visit [our documentation](https://fairdatapoint.readthedocs.io/en/latest/openrefine/setup.html#configuration).
## Development
You are required to have Maven installed with other necessary tools for building Java (see [OpenRefine - Documentation for Developers](https://github.com/OpenRefine/OpenRefine/wiki/Documentation-For-Developers)):
```console
$ mvn clean compile
```
To create a distribution ZIP/TGZ files, run:
```console
$ mvn package
```
After issuing these commands, prepared ZIP/TGZ files should be located in `target` directory according to the Maven output.
We maintain a [CHANGELOG](CHANGELOG.md), you should also take a look at our [Contributing guidelines](CONTRIBUTING.md) and
[Code of Conduct](CODE_OF_CONDUCT.md).
### Build with Docker
You can also use provided [multistage](https://docs.docker.com/develop/develop-images/multistage-build/) `Dockerfile`
that builds the extension and then runs it in OpenRefine. Therefore, you don't need
any Java dependencies locally, just Docker:
```console
$ docker build -t openrefine-metadata:local-tag .
$ docker run -p 3333:3333 openrefine-metadata:local-tag
```
Then open [http://localhost:3333](http://localhost:3333) in your favorite web browser.
You can also build a Docker image with specific OpenRefine version using argument (see [Dockerfile](Dockerfile) for details):
```console
docker build . -t openrefine-metadata:local-or3.2 --build-arg OPENREFINE_VERSION=3.2
```
Note that we are using Java SE 8 (LTS), concretely OpenJDK 8 in Docker images (to comply with OpenRefine requirements).
## License
This project is licensed under MIT license - see the [LICENSE](LICENSE) file for more information.
<?xml version="1.0" encoding="UTF-8"?>
<!--
The MIT License
Copyright © 2019 FAIR Data Team
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>solutions.fairdata.openrefine</groupId>
<artifactId>metadata</artifactId>
<version>1.5.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>OpenRefine - FAIR Metadata extension</name>
<description>Extension for managing metadata in FAIR way</description>
<url>https://github.com/FAIRDataTeam/OpenRefine-metadata-extension</url>
<inceptionYear>2019</inceptionYear>
<organization>
<name>FAIR Data Team</name>
<url>https://github.com/FAIRDataTeam</url>
</organization>
<licenses>
<license>
<name>MIT</name>
<url>https://github.com/FAIRDataTeam/OpenRefine-metadata-extension/LICENSE</url>
</license>
</licenses>
<developers>
<developer>
<name>Marek Suchánek</name>
<email>marek.suchanek@fit.cvut.cz</email>
<url>https://suchanek.cloud</url>
<organization>FIT CTU in Prague</organization>
</developer>
</developers>
<properties>
<!-- Project -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!-- Maven -->
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<!-- Core - local -->
<openrefine.version>3.3</openrefine.version>
<servlet.version>2.5</servlet.version>
<butterfly.version>1.0.2</butterfly.version>
<slf4j.version>1.7.18</slf4j.version>
<!-- Core - other -->
<rdf4j.version>3.0.3</rdf4j.version>
<jackson.version>2.10.1</jackson.version>
<commons.net.version>3.6</commons.net.version>
<lombok.version>1.18.10</lombok.version>
<!-- Test -->
<test.junit-jupiter.api.version>5.5.2</test.junit-jupiter.api.version>
<test.junit-jupiter.engine.version>5.5.2</test.junit-jupiter.engine.version>
<!-- Plugins -->
<maven.build-helper.version>3.0.0</maven.build-helper.version>
<maven.dependency.version>3.1.1</maven.dependency.version>
<maven.clean.version>3.1.0</maven.clean.version>
<maven.assembly.version>3.1.1</maven.assembly.version>
<maven.spotbug.version>3.1.12</maven.spotbug.version>
<maven.tidy.version>1.1.0</maven.tidy.version>
<maven.license.version>3.0</maven.license.version>
</properties>
<dependencies>
<dependency>
<groupId>org.openrefine</groupId>
<artifactId>main</artifactId>
<version>${openrefine.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>${servlet.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>edu.mit.simile</groupId>
<artifactId>butterfly</artifactId>
<version>${butterfly.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>rdf4j-query</artifactId>
<version>${rdf4j.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>rdf4j-rio-api</artifactId>
<version>${rdf4j.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>rdf4j-rio-turtle</artifactId>
<version>${rdf4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>rdf4j-repository-http</artifactId>
<version>${rdf4j.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>rdf4j-repository-api</artifactId>
<version>${rdf4j.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>${commons.net.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
<!-- TEST -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${test.junit-jupiter.api.version}</version>
<scope>test</scope>
</dependency>