watzon
/
wordpos
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move web dict files & samples to own repo

This commit is contained in:
moos 2019-12-31 10:58:24 -08:00
parent 3277dcd3fe
commit 7dd8072d26
14 changed files with 1022 additions and 1221 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -5,3 +5,4 @@ node_modules
build build
dict dict
dist dist
*.tgz

1
.npmignore
View File

@ -2,3 +2,4 @@ node_modules
.project .project
.idea .idea
*.iml *.iml
*.tgz

1
.travis.yml
View File

@ -1,5 +1,6 @@
language: node_js language: node_js
node_js: node_js:
- '13'
- '12' - '12'
- '11' - '11'
- '10' - '10'

57
CHANGELOG.md Normal file
View File

@ -0,0 +1,57 @@
**2.0.0**
- Support for running wordpos in the **browser** (no breaking change for node environment)
- Dropped support for node 4.x.
1.2.0 (tagged v1.2.0)
- Fix `new Buffer()` deprecation warning.
- Fix npm audit vulnerabilities
1.1.6
- Fix #25 rand().then with no args
1.1.5
- rollback 1.1.4 changes. Fix is made in [wordnet-db](https://github.com/moos/wordnet-db).
1.1.4
- temporary fix for #19 issue with npm@5
1.1.2
- Fix DeprecationWarning for node 7.x (1.1.1)
- Fix occasional error for large offsets during seek
1.1.0
- added seek() method
- added lexName property
1.0.1
- Removed npm dependency on Natural. Certain modules are included in /lib.
- Add support for ES6 Promises.
- Improved data file reads for up to **5x** performance increase compared to previous version.
- Tests are now [mocha](https://mochajs.org/)-based with [chai](http://chaijs.com/) assert interface.
0.1.16
- Changed dependency to wordnet-db (renamed from WNdb)
0.1.15
- Added `syn` (synonym) and `exp` (example) CLI commands.
- Fixed `rand` CLI command when no start word given.
- Removed -N, --num CLI option. Use `wordpos rand [N]` to get N random numbers.
- Changed CLI option -s to -w (include stopwords).
0.1.13
- Fix crlf issue for command-line script
0.1.12
- fix stopwords not getting excluded when running with CLI
- added 'stopwords' CLI *command* to show list of stopwords
- CLI *option* --stopword now renamed to --withStopwords
0.1.10
- rand functionality added
0.1.6
- added command line tool
0.1.4
- added fast index

100
README.md
View File

@ -294,41 +294,9 @@ wordpos.isVerb('fish', console.log)
``` ```
Note that callback receives full arguments (including profile, if enabled), while the Promise receives only the result of the call. Also, beware that exceptions in the _callback_ will result in the Promise being _rejected_ and caught by `catch()`, if provided. Note that callback receives full arguments (including profile, if enabled), while the Promise receives only the result of the call. Also, beware that exceptions in the _callback_ will result in the Promise being _rejected_ and caught by `catch()`, if provided.
## Running inside the browsers ## Running inside the browsers?
v2.0 introduces the capability of running wordpos in the browser. The dictionary files are optimized for fast access (lookup by lemma), but they must be fetched, parsed and loaded into browser memory. The files are loaded on-demand (unless the option `preload: true` is given). See [wordpos-web](https://github.com/moos/wordpos-web).
The dict files can be served locally or from CDN (see [samples/cdn](samples/cdn/) for code, or [see it in action](https://moos.github.io/wordpos/cdn)). Include the following scripts in your `index.html`:
```html
<script src="wordpos/dist/wordpos.min.js"></script>
<script>
let wordpos = new WordPOS({
// preload: true,
dictPath: '/wordpos/dict',
profile: true
});
wordpos.getAdverbs('this is is lately a likely tricky business this is')
.then(res => {
console.log(res); // ["lately", "likely"]
});
</script>
```
Above assumes wordpos is installed to the directory `./wordpos`. `./wordpos/dict` holds the index and data WordNet files generated for the web in a postinstall script.
See [samples/self-hosted](samples/self-hosted/).
To run the samples locally, install [parcel](https://github.com/parcel-bundler/parcel) if you don't already have it (`npm i -g parcel`), then:
```bash
$ npm run start-self
Server running at http://localhost:1234
...
$ npm run start-cdn
Server running at http://localhost:1234
...
```
and open your browser to that url.
## Fast Index (node) ## Fast Index (node)
@ -338,7 +306,7 @@ Fast index improves performance **30x** over Natural's native methods. See blog
As of version 1.0, fast index is always on and cannot be turned off. As of version 1.0, fast index is always on and cannot be turned off.
## Command-line: CLI ## Command-line (CLI) usage
For CLI usage and examples, see [bin/README](bin). For CLI usage and examples, see [bin/README](bin).
@ -353,67 +321,11 @@ See [bench/README](bench).
## Changes ## Changes
See [CHANGELOG](./CHANGELOG.md).
**2.0.0**
- Support for running wordpos in browser (no breaking change for node environment)
- Dropped support for node 4.x.
1.2.0
- Fix `new Buffer()` deprecation warning.
- Fix npm audit vulnerabilities
1.1.6
- Fix #25 rand().then with no args
1.1.5
- rollback 1.1.4 changes. Fix is made in [wordnet-db](https://github.com/moos/wordnet-db).
1.1.4
- temporary fix for #19 issue with npm@5
1.1.2
- Fix DeprecationWarning for node 7.x (1.1.1)
- Fix occasional error for large offsets during seek
1.1.0
- added seek() method
- added lexName property
1.0.1
- Removed npm dependency on Natural. Certain modules are included in /lib.
- Add support for ES6 Promises.
- Improved data file reads for up to **5x** performance increase compared to previous version.
- Tests are now [mocha](https://mochajs.org/)-based with [chai](http://chaijs.com/) assert interface.
0.1.16
- Changed dependency to wordnet-db (renamed from WNdb)
0.1.15
- Added `syn` (synonym) and `exp` (example) CLI commands.
- Fixed `rand` CLI command when no start word given.
- Removed -N, --num CLI option. Use `wordpos rand [N]` to get N random numbers.
- Changed CLI option -s to -w (include stopwords).
0.1.13
- Fix crlf issue for command-line script
0.1.12
- fix stopwords not getting excluded when running with CLI
- added 'stopwords' CLI *command* to show list of stopwords
- CLI *option* --stopword now renamed to --withStopwords
0.1.10
- rand functionality added
0.1.6
- added command line tool
0.1.4
- added fast index
License License
------- -------
https://github.com/moos/wordpos
Copyright (c) 2012-2020 mooster@42at.com
(The MIT License) (The MIT License)
Copyright (c) 2012-2019 mooster@42at.com

1
docs
View File

@ -1 +0,0 @@
samples/

2424
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

67
package.json
View File

@ -1,6 +1,6 @@
{ {
"name": "wordpos", "name": "wordpos",
"version": "2.0.0-beta.2", "version": "2.0.0-beta.10",
"description": "wordpos is a set of part-of-speech utilities for Node.js & browser using the WordNet database.", "description": "wordpos is a set of part-of-speech utilities for Node.js & browser using the WordNet database.",
"author": "Moos <mooster@42at.com>", "author": "Moos <mooster@42at.com>",
"keywords": [ "keywords": [
@ -17,45 +17,40 @@
"node": ">=6" "node": ">=6"
}, },
"files": [ "files": [
"bench",
"bin", "bin",
"dict",
"dist", "dist",
"lib",
"src", "src",
"scripts",
"test", "test",
"!test/dict", "!test/dict",
"tools" "tools"
], ],
"bin": "./bin/wordpos-cli.js", "bin": "./bin/wordpos-cli.js",
"dependencies": { "dependencies": {
"commander": "^2.0.0", "commander": "^2.20.3",
"symlink-dir": "1.1.3",
"underscore": ">=1.3.1", "underscore": ">=1.3.1",
"wordnet-db": "^3.1.11" "wordnet-db": "^3.1.14"
}, },
"devDependencies": { "devDependencies": {
"@babel/core": "^7.0.0", "@babel/core": "^7.7.7",
"@babel/plugin-proposal-class-properties": "^7.0.0", "@babel/plugin-proposal-class-properties": "^7.7.4",
"@babel/plugin-proposal-decorators": "^7.0.0", "@babel/plugin-proposal-decorators": "^7.7.4",
"@babel/plugin-proposal-do-expressions": "^7.0.0", "@babel/plugin-proposal-do-expressions": "^7.7.4",
"@babel/plugin-proposal-export-default-from": "^7.0.0", "@babel/plugin-proposal-export-default-from": "^7.7.4",
"@babel/plugin-proposal-export-namespace-from": "^7.0.0", "@babel/plugin-proposal-export-namespace-from": "^7.7.4",
"@babel/plugin-proposal-function-sent": "^7.0.0", "@babel/plugin-proposal-function-sent": "^7.7.4",
"@babel/plugin-proposal-json-strings": "^7.0.0", "@babel/plugin-proposal-json-strings": "^7.7.4",
"@babel/plugin-proposal-logical-assignment-operators": "^7.0.0", "@babel/plugin-proposal-logical-assignment-operators": "^7.7.4",
"@babel/plugin-proposal-nullish-coalescing-operator": "^7.0.0", "@babel/plugin-proposal-nullish-coalescing-operator": "^7.7.4",
"@babel/plugin-proposal-numeric-separator": "^7.0.0", "@babel/plugin-proposal-numeric-separator": "^7.7.4",
"@babel/plugin-proposal-optional-chaining": "^7.0.0", "@babel/plugin-proposal-optional-chaining": "^7.7.5",
"@babel/plugin-proposal-pipeline-operator": "^7.0.0", "@babel/plugin-proposal-pipeline-operator": "^7.7.7",
"@babel/plugin-proposal-throw-expressions": "^7.0.0", "@babel/plugin-proposal-throw-expressions": "^7.7.4",
"@babel/plugin-syntax-dynamic-import": "^7.0.0", "@babel/plugin-syntax-dynamic-import": "^7.7.4",
"@babel/plugin-syntax-import-meta": "^7.0.0", "@babel/plugin-syntax-import-meta": "^7.7.4",
"@babel/preset-env": "^7.0.0", "@babel/preset-env": "^7.7.7",
"@babel/register": "^7.0.0", "@babel/register": "^7.7.7",
"babel-plugin-dynamic-import-node": "^2.2.0", "babel-plugin-dynamic-import-node": "^2.3.0",
"chai": "^4.0.2", "chai": "^4.2.0",
"mini-bench": "^1.0.0", "mini-bench": "^1.0.0",
"mocha": "^5.2.0" "mocha": "^5.2.0"
}, },
@ -66,20 +61,14 @@
"main": "./src/node/index.js", "main": "./src/node/index.js",
"browser": "./src/browser/index.js", "browser": "./src/browser/index.js",
"scripts": { "scripts": {
"postinstall": "npm run postinstall-web && npm run postinstall-node", "postinstall": "node tools/stat.js --no-stats index.adv index.adj index.verb index.noun",
"postinstall-node": "node tools/stat.js --no-stats index.adv index.adj index.verb index.noun",
"postinstall-web": "node scripts/makeJsonDict.js index data",
"build": "parcel build --detailed-report -d dist -o wordpos.min.js --global WordPOS -t browser src/browser/index.js", "build": "parcel build --detailed-report -d dist -o wordpos.min.js --global WordPOS -t browser src/browser/index.js",
"postbuild": "sed -i 's/ES6_IMPORT/import/' dist/wordpos.min.js", "postbuild": "sed -i 's/ES6_IMPORT/import/' dist/wordpos.min.js && node tools/banner.js | cat - dist/wordpos.min.js > _tmp_ && mv _tmp_ dist/wordpos.min.js",
"prepare": "npm run build",
"pretest": "node tools/makeJsonDict.js index data",
"test": "npm run test-node && npm run test-browser", "test": "npm run test-node && npm run test-browser",
"test-node": "mocha test", "test-node": "mocha test",
"test-browser": "mocha test/wordpos_test --require @babel/register", "test-browser": "mocha test/wordpos_test --require @babel/register"
"prestart": "symlink-dir dict samples/self-hosted/dict",
"start": "npm run build && http-server",
"prestart-dev": "rm -rf build && mkdir build && symlink-dir dict build/dict && cp samples/main.js build/main.txt",
"start-dev": "npm run start-self -- -d build",
"start-self": "parcel samples/self-hosted/index.html",
"start-cdn": "parcel samples/cdn/index.html"
}, },
"license": "MIT" "license": "MIT"
} }

53
samples/cdn/index.html
View File

@ -1,53 +0,0 @@
<!doctype html>
<html>
<head>
<meta http-equiv="Content-Security-Policy" content="script-src https: http: 'unsafe-inline' 'unsafe-eval'">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/github.min.css" />
<script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
<script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/languages/javascript.min.js"></script>
<!-- cdn source -->
<script src="https://unpkg.com/wordpos@2.0.0-beta/dist/wordpos.min.js"></script>
<script>
let wordpos = window.wordpos = new WordPOS({
// preload: true,
dictPath: 'https://unpkg.com/wordpos@2.0.0-beta/dict',
profile: true,
// stopwords: false
});
</script>
<script src="../main.js" name="main"></script>
<style>
pre {
padding: 2em;
display: block;
}
</style>
</head>
<body>
<h1>WordPOS CDN sample</h1>
Open console to see results.
<pre><code> </code></pre>
<script>
var el = document.querySelector('code');
if (fetch) {
fetch('../main.js')
.then(res => res.text())
.then(txt => {
el.innerText = txt;
window.hljs && hljs.initHighlightingOnLoad();
});
} else {
el.innerHTML = 'Open <a href="../main.js">main.js</a>.';
}
</script>
</body>
</html>

9
samples/index.html
View File

@ -1,9 +0,0 @@
<!doctype html>
<html>
<head>
<meta-ignore-me http-equiv="refresh" content="0; URL='cdn'" />
</head>
<body>
See <a href="cdn">WordPOS CDN</a> sample.
</body>
</html>

32
samples/main.js
View File

@ -1,32 +0,0 @@
let assertLikely = (r) => {
console.assert(r.def === 'with considerable certainty');
console.assert(r.pos === 'r');
console.assert(r.synsetOffset === '00139421');
};
console.group('Likely');
wordpos.isAdverb('likely').then(res => console.assert(res));
wordpos.isAdverb('likely', (res, ...profile) => console.log('callback with profile', res, profile));
wordpos.getAdverbs('this is is lately a likely tricky business this is')
.then(res => {
let expect = {lately: 1, likely: 1};
console.log('getAdverbs:', res);
console.assert(res[0] in expect); // NOTE: order is NOT gauranteed!
console.assert(res[1] in expect);
});
wordpos.lookupAdverb('likely')
.then(res => {
console.log('lookupAdverb:', res[0]);
assertLikely(res[0]);
});
// wordpos.lookup('likely').then(res, console.log('lookup ===', res))
wordpos.seek('00139421', 'r')
.then(res => {
console.log('seek:', res);
assertLikely(res);
});
setTimeout(() => console.groupEnd('Likely'), 1000);

52
samples/self-hosted/index.html
View File

@ -1,52 +0,0 @@
<!doctype html>
<html>
<head>
<meta http-equiv="Content-Security-Policy" content="script-src https: http: 'unsafe-inline' 'unsafe-eval'">
<title>Wordpos in the browser</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/github.min.css" />
<script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
<script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/languages/javascript.min.js"></script>
<script src="/dist/wordpos.min.js"></script>
<script>
let wordpos = window.wordpos = new WordPOS({
// preload: true,
dictPath: '/samples/self-hosted/dict',
profile: true,
// stopwords: false
});
</script>
<script src="../main.js" name="main"></script>
<style>
pre {
padding: 2em;
display: block;
}
</style>
</head>
<body>
<h1>Self-hosted WordPOS sample</h1>
Open console to see results.
<pre><code> </code></pre>
<script>
var el = document.querySelector('code');
if (fetch) {
fetch('../main.js')
.then(res => res.text())
.then(txt => {
el.innerText = txt;
window.hljs && hljs.initHighlightingOnLoad();
});
} else {
el.innerHTML = 'Open <a href="../main.js">main.js</a>.';
}
</script>
</body>
</html>

22
tools/banner.js Normal file
View File

@ -0,0 +1,22 @@
const pkg = require('../package.json');
const wndb = require('wordnet-db');
const copyright = `/*!
Copyright (c) 2012-2020 mooster@42at.com
https://github.com/moos/wordpos (The MIT License)
Princeton University "About WordNet." WordNet (https://wordnet.princeton.edu/). Princeton University. 2010.
${pkg.name} v${pkg.version}
wordnet-db v${wndb.libVersion}
WordNet DB version ${wndb.version}
*/
`;
module.exports = {
copyright
};
if (require.main === module) {
process.stdout.write(copyright);
}

33
scripts/makeJsonDict.js → tools/makeJsonDict.js
View File

@ -5,27 +5,32 @@
* exported JSON format with lemma as the key. * exported JSON format with lemma as the key.
*/ */
let fs = require('fs'); const fs = require('fs');
let path = require('path'); const path = require('path');
const pkg = require('../package.json');
let outPath = './dict'; // browser-use files const wndb = require('wordnet-db');
let testPath = './test/dict'; // mocha files in CJS format
let posExt = ['adj', 'adv', 'noun', 'verb'];
let dictRoot = require('wordnet-db').path; // source files
const outPath = './dict'; // browser-use files
const testPath = './test/dict'; // mocha files in CJS format
const testOpt = '--no-test'; // don't do test format
const posExt = ['adj', 'adv', 'noun', 'verb'];
const dictRoot = wndb.path; // source files
const copyright = require('./banner').copyright;
const fileTypes = { const fileTypes = {
data: true, data: true,
index: true index: true
}; };
const [,, ...args] = process.argv; const [,, ...args] = process.argv;
if (!args.length || args.filter(p => !(p in fileTypes)).length) { if (!args.length || args.filter(p => p !== testOpt && !(p in fileTypes)).length) {
console.log('Converts wordnet-db index & data files to JSON format for use in the browser.'); console.log('Converts wordnet-db index & data files to JSON format for use in the browser.');
console.log('\nUsage: makeJsonDict.js index|data'); console.log(`\nUsage: makeJsonDict.js index|data [${testOpt}]`);
process.exit(1); process.exit(1);
} }
const doTest = !args.includes(testOpt);
if (!doTest) args.splice(args.indexOf(testOpt));
function uniq(arr) { function uniq(arr) {
return arr.filter((v, i) => arr.indexOf(v) === i); return arr.filter((v, i) => arr.indexOf(v) === i);
} }
@ -42,7 +47,7 @@ const ensurePath = (path) => {
}; };
ensurePath(outPath); ensurePath(outPath);
ensurePath(testPath); if (doTest) ensurePath(testPath);
function processFile(name) { function processFile(name) {
@ -75,11 +80,11 @@ function processFile(name) {
console.time(' write'); console.time(' write');
let text = JSON.stringify(obj); let text = JSON.stringify(obj);
fs.writeFileSync(path.resolve(outPath, name + '.' + pos + '.js'), fs.writeFileSync(path.resolve(outPath, name + '.' + pos + '.js'),
'export default ' + text); copyright + 'export default ' + text);
// also write for mocha tests // also write for mocha tests
fs.writeFileSync(path.resolve(testPath, name + '.' + pos + '.js'), if (doTest) fs.writeFileSync(path.resolve(testPath, name + '.' + pos + '.js'),
'module.exports.default = ' + text); copyright + 'module.exports.default = ' + text);
console.timeEnd(' write'); console.timeEnd(' write');
} }