name | Readwise |
description | Sync Highlights from Readwise. |
icon | menu_book |
setting | Readwise Access Token |
setting | Base tag for Readwise notes (Default: library) |
setting | Import discarded highlights ("true" or "false". Default: false) |
setting | Readwise dashboard note title (default: Readwise Library Dashboard) |
setting | Save authors as tags ("true" or "false". Default: false) |
setting | Highlight sort order ("newest" or "oldest". Default: newest) |
instructions | tl; dr Install plugin. Provide Readwise API key within API settings. Choose "Sync all":  The Readwise plugin allows you to keep your Amplenote notebook synchronized with the reading that you have connected with your Readwise account. To get started syncing from Readwise: 1. Install this plugin (i.e., by clicking the "Install" button when you are logged in to Amplenote) 2. Visit your plugin settings (Settings -> Plugins). Click the settings icon on your Readwise plugin to open settings for Readwise. 3. Paste your Readwise API key into the "Readwise API key" field in its plugin settings. You can fetch your Readwise API key from this page on Readwise. 4. Change any other desired plugin settings, for example if there is a specific tag structure you want to use for where your highlights. Save your plugin settings and return to your notes. 5. Open the note that you want to serve as your index for your books. Then hit Ctrl-O or Cmd-O and type in "Sync all".At this point, you will see that your notebook begins to import individual notes that correspond to books you have read. Each book note will be tagged in the structure you configured in plugin settings. If you choose "Sync all" again later, the plugin will only update highlights for books that have received new highlights since your previous sync. This individual book notes play host to your full list of Readwise highlights and notes:  Each book will store up to 5,000 highlights you made. If you make additional highlights within the book after your initial sync, you can run "Readwise: Sync All" again, OR you can visit the book's note directly (all notes will be tagged with the base tag you choose within your plugin settings, or the default "library" tag if nothing) and pick "Readwise: Sync this book" from the plugin choices in the note option menu. When choosing which highlights to sync in from a book, the "Updated at" time from the book note's "Sync history" section (bottom of note) is used to ensure we don't double-grab books/highlights. If you would like to re-grab the highlights for a book, you will need to delete both the highlight(s) that are to be re-imported, and the "Sync history" update dates. ⚡ Development on this plugin is ongoing. You can follow what's being developed next at its GitHub repo. |
repo: https://github.com/alloy-org/readwise
linkVersion history
November 16th, 2023
Add clearer highlight heading meanings (eg 2023 (part 2) instead of just 2023 2)
November 13th, 2023
Fix formatting issues when syncing the second time
October 3rd, 2023:
Limit table image width to 200ps in the Readwise Library Dashboard
October 2nd, 2023
Don't offer to sync a book unless we are in a book note
Remove note options for "Sync all" and "Sync only", add these as app options instead
August 22nd, 2023:
Better logging
August 21st, 2023:
Fix: more cases where books have null date fields
August 18th, 2023:
Fix: handled case where some Readwise books have null/undefined fields (author, category, update time or source)
Fix: add custom tag to the dashboard note
Don't save authors as tags by default
August 17th, 2023:
Add more logging to catch and document rare TypeErrors that result in empty dashboard notes
August 15th, 2023:
Fix error that appears in some cases where a book contains long (>5000 characters) highlights
July 20th, 2023:
Better handling of 400+ libraries (details)
Fix crash on importing books with a large number of highlights
"Sync only..." option now available, which syncs only a chosen Category (eg. books, or articles or tweets etc.)
June 27, 2023: Book index note split into years for more manageable table sizes & faster parsing for users with 100+ books.
June 15, 2023. Updated plugin to work better with large Readwise libraries. Don't throw alert message when we get a bad response, just swallow it and try again in some seconds. Add option for whether or not to create tags for authors. Put new books at the top of table if their last update date is newer than the current book atop table. Only grab books older & newer than those that have already been grabbed.
June 6, 2023. Added plugin icon.
June 4, 2023
Introduce a new setting field for the dashboard note title, so "Sync all" can be chosen from any note and end up locating the index of previously imported Readwise sources in a consistent note. For existing users, paste the title of your existing readwise index note into the settings field "Readwise dashboard note title", or rename your Readwise dashboard note to be "Readwise Library Dashboard"
Updated book index table such that it is refreshed on each "Sync all" run. Add count of highlights to each entry in the book index.
Use highlight identifiers to avoid double-syncing highlights with markdown formatting that doesn't match Amplenote markdown formatting
Add links in book details to author's tag, and to Kindle source. Also add field for the total count of highlights made for book.
Fix notes with more than 100,000 characters of highlights creating hard fail during sync
Fetch up to 1,000 books by default on first page instead of 100
Fix reversion to 3 books imported.
June 3, 2023. Added links to each individual highlight within Kindle.
June 2, 2023. V1 release. Create an index note with "Sync all," and create notes for each book. Allow books to be individually synced via "Sync book" note option. Allow book notes to have a default tag applied, and apply a tag for author name as well.
linkTODOs
Don't modify existing highlights, even if they contain changes
Let users choose which columns to sync
Let users choose which highlight properties to sync
Tag the dashboard with "readwise/overview" instead
Support syncing individual document-level tags
Support syncing highlight-level tags
linkCode block
// Javascript updated 11/16/2023, 5:01:43 PM by Amplenote Plugin Builder from source code within "https://github.com/alloy-org/readwise"
{
// TODO: handle abort execution
// TODO: add conditions to plugin actions
constants: {
defaultBaseTag: "library",
dashboardBookListTitle: "Readwise Book List",
defaultDashboardNoteTitle: "Readwise Library Dashboard",
defaultHighlightSort: "newest",
dashboardLibraryDetailsHeading: "Library Details",
dashDetails: {
lastSyncedAt: "Last synced at",
firstUpdated: "Oldest update synced in",
lastUpdated: "Next sync for content updated after",
booksImported: "Readwise books imported into table",
booksReported: "Book count reported by Readwise",
},
maxBookLimitInMemory: 20,
maxReplaceContentLength: 100000, // Empirically derived
maxHighlightLimit: 5000,
maxTableBooksPerSection: 20,
maxBookHighlightsPerSection: 10,
maxBookLimit: 500,
noHighlightSectionLabel: "(No highlights yet)",
rateLimit: 20, // Max requests per minute (20 is Readwise limit for Books and Highlights APIs)
readwiseBookDetailURL: bookId => `https://readwise.io/api/v2/books/${ bookId }`,
readwiseBookIndexURL: "https://readwise.io/api/v2/books",
readwiseExportURL: "https://readwise.io/api/v2/export",
readwiseHighlightsIndexURL: "https://readwise.io/api/v2/highlights",
readwisePageSize: 1000, // Highlights and Books both claim they can support page sizes up to 1000 so we'll take them up on that to reduce number of requests we need to make
sectionRegex: /^#+\s*([^#\n\r]+)/gm,
settingAuthorTag: "Save authors as tags (\"true\" or \"false\". Default: false)",
settingDateFormat: "Date format (default: en-US)",
settingDiscardedName: "Import discarded highlights (\"true\" or \"false\". Default: false)",
settingSortOrderName: "Highlight sort order (\"newest\" or \"oldest\". Default: newest)",
settingTagName: "Base tag for Readwise notes (Default: library)",
sleepSecondsAfterRequestFail: 10,
updateStringPreface: "- Highlights updated at: ",
unsortedSectionTitle: "Books pending sort",
},
appOption: {
/*******************************************************************************************
* Fetches all books found in Readwise. Creates a note per book.
*/
"Sync all": async function (app) {
this._initialize(app);
this._useLocalNoteContents = true;
await this._syncAll(app);
},
/*******************************************************************************************
* Fetches all items of a certain category. Creates a note per item.
*/
"Sync only...": async function(app) {
this._initialize(app);
this._useLocalNoteContents = true;
await this._syncOnly(app);
},
},
noteOption: {
/*******************************************************************************************
* Syncs newer highlights for an individual book.
* Fails if the note title doesn't match the required template.
*/
"Sync this book": {
run: async function(app, noteUUID) {
this._initialize(app);
this._useLocalNoteContents = true;
await this._syncThisBook(app, noteUUID);
},
/*
* Only show the option to sync a book if the note title has the expected format and tag applied
*/
check: async function(app, noteUUID) {
const noteObject = await app.findNote({uuid: noteUUID});
const noteTitle = noteObject.name;
const bookTitleRegExp = new RegExp(".*\(ID #[0-9]+\)");
if (!bookTitleRegExp.test(noteTitle)) return false;
for (const tag of noteObject.tags) {
if (tag.startsWith(app.settings[this.constants.settingTagName] || this.constants.defaultBaseTag)) return true;
}
return false;
},
}
},
/*******************************************************************************************/
/* Main entry points
/*******************************************************************************************/
async _syncAll(app, categoryFilter) {
console.log("Starting sync all", new Date());
try {
const dashboardNoteTitle = app.settings[`Readwise dashboard note title (default: ${ this.constants.defaultDashboardNoteTitle })`] ||
this.constants.defaultDashboardNoteTitle;
if (this._abortExecution) app.alert("_abortExecution is true")
[ this._forceReprocess, this._dateFormat ] = await app.prompt("Readwise sync options", {
inputs: [
{ label: "Force reprocess of all book highlights?", type: "select", options:
[
{ value: "false", label: `No (uses "Last updated" dates to sync only new)` },
{ value: "true", label: `Yes (slower, uses more quota)` }
]
},
{ label: "Date format", type: "select", options:
[
{ value: "default", label: `Current default (${ app.settings[this.constants.settingDateFormat] || "en-US" })` },
{ value: "en-US", label: "en-US (English - United States)" },
{ value: "en-GB", label: "en-GB (English - United Kingdom)" },
{ value: "de-DE", label: "de-DE (German - Germany)" },
{ value: "fr-FR", label: "fr-FR (French - France)" },
{ value: "es-ES", label: "es-ES (Espanol - Spain)" },
{ value: "it-IT", label: "it-IT (Italian - Italy)" },
{ value: "ja-JP", label: "ja-JP (Japanese - Japan)" },
{ value: "ko-KR", label: "ko-KR (Korean - Korea)" },
{ value: "pt-PT", label: "pt-PT (Portuguese - Portugal)" },
{ value: "pt-BR", label: "pt-BR (Portuguese - Basil)" },
{ value: "zh-CN", label: "zh-CN (Chinese - China)" },
{ value: "zh-TW", label: "zh-TW (Chinese - Taiwan)" },
]
},
]
});
// Ensure that dashboardNote exists in a state where await this._inlined_amplenote_rw_js__noteContent(dashboardNote) can be called on it
const baseTag = app.settings[this.constants.settingTagName] || this.constants.defaultBaseTag;
let dashboardNote = await app.findNote({ name: dashboardNoteTitle, tag: baseTag});
if (dashboardNote) {
console.log("Found existing dashboard note", dashboardNote, "for", dashboardNoteTitle);
dashboardNote = await app.notes.find(dashboardNote.uuid);
} else {
console.log("Creating dashboard note anew");
dashboardNote = await app.notes.create(dashboardNoteTitle, [ baseTag ]);
}
// Move to existing or new dashboard note
if (app.context.noteUUID !== dashboardNote.uuid) {
let origin;
try {
origin = window.location.origin.includes("localhost") ? "http://localhost:3000" : window.location.origin.replace("plugins", "www");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line (141)`));
}
}
const navigateUrl = `${ origin }/notes/${ dashboardNote.uuid }`;
await app.navigate(navigateUrl);
}
let bookCount = 0;
await this._inlined_dashboard_js__migrateBooksToSections(app, dashboardNote);
let dashboardNoteContents = await this._inlined_amplenote_rw_js__noteContent(dashboardNote);
const details = this._inlined_dashboard_js__loadDetails(this._inlined_amplenote_rw_js__sectionContent(dashboardNoteContents, this.constants.dashboardLibraryDetailsHeading));
if (!dashboardNoteContents.includes(this.constants.dashboardLibraryDetailsHeading)) {
await this._inlined_amplenote_rw_js__insertContent(dashboardNote, "# " + this.constants.dashboardLibraryDetailsHeading + "\n");
}
if (!dashboardNoteContents.includes(this.constants.dashboardBookListTitle)) {
// Add a header to the dashboard note if it doesn't exist
// Edits dashboard note
await this._inlined_amplenote_rw_js__insertContent(dashboardNote, `# ${ this.constants.dashboardBookListTitle }\n`, { atEnd: true });
}
const updateThrough = details.lastUpdated;
let dateFilter = null;
if (updateThrough) {
dateFilter = new Date(Date.parse(updateThrough));
dateFilter = dateFilter.toISOString().slice(0, -1) + 'Z';
console.log("Looking for results after", updateThrough, "submitting as", dateFilter);
}
let dashboard = await this._inlined_markdown_js__sectionsFromMarkdown(dashboardNote, this.constants.dashboardBookListTitle, this._inlined_markdown_js__tableFromMarkdown);
dashboard = this._inlined_data_structures_js__groupByValue(dashboard,
item => {
return this._inlined_dashboard_js__sectionNameFromLastHighlight(item.Updates);
},
);
let readwiseBookCount = await this._inlined_dashboard_js__getReadwiseBookCount(app);
if (readwiseBookCount) {
await this._inlined_dashboard_js__updateDashboardDetails(app, dashboard, details, { bookCount: readwiseBookCount });
}
for await (const readwiseBook of this._inlined_readwise_js__readwiseFetchBooks(app, {dateFilter, categoryFilter})) {
if (this._abortExecution) break;
if (!readwiseBook) continue;
if (bookCount >= this.constants.maxBookLimit) break;
const bookNote = await this._inlined_books_js__ensureBookNote(app, readwiseBook, dashboardNote);
const bookObject = this._inlined_books_js__bookObjectFromReadwiseBook(app, readwiseBook, bookNote.uuid);
// Edits dashboard object
await this._inlined_dashboard_js__ensureBookInDashboardNoteTable(app, dashboard, bookObject);
// Edits book note
const success = await this._inlined_books_js__syncBookHighlights(app, bookNote, readwiseBook.id, { readwiseBook });
if (success) bookCount += 1;
}
// Edits dashboard note
// Let's load it again just in case it was deleted in previous flush notes
// TODO: fixme
await this._inlined_amplenote_rw_js__noteContent(dashboardNote);
let tableRowCount = Object.values(dashboard).reduce((total, curr) => total + curr.length, 0);
await this._inlined_dashboard_js__updateDashboardDetails(app, dashboard, details, {tableRowCount});
let markdownDetails = this._inlined_dashboard_js__writeDetails(details);
await this._inlined_amplenote_rw_js__replaceContent(dashboardNote, this.constants.dashboardLibraryDetailsHeading, markdownDetails);
console.log(JSON.stringify((dashboard)));
await this._inlined_dashboard_js__writeDashboard(app, dashboard, dashboardNote);
if (this._useLocalNoteContents) {
await this._inlined_amplenote_rw_js__flushLocalNotes(app);
}
if (this._abortExecution) {
await app.alert(`✅️ ${ bookCount } book${ bookCount === "1" ? "" : "s" } refreshed before canceling sync.`);
} else {
await app.alert(`✅ ${ bookCount } book${ bookCount === "1" ? "" : "s" } fetched & refreshed successfully!`);
}
} catch (error) {
if (this._testEnvironment) {
throw(error);
} else {
console.trace();
await app.alert(String(error));
this._abortExecution = true;
}
} finally {
this._useLocalNoteContents = false;
}
},
/*******************************************************************************************/
async _syncThisBook(app, noteUUID) {
try {
const currentNote = await app.notes.find(noteUUID);
const noteTitle = currentNote.name;
// Check if the note title is of the format "Readwise: {book title} {book id}"
const titleRegex = /ID\s?#([\d]+)/;
const match = noteTitle.match(titleRegex);
if (!match) {
throw new Error("The note title format is incorrect. It should contain an 'ID' designator, like 'ID: #123', in the title");
}
// Import all (new) highlights from the book
const bookId = match[1];
const success = await this._inlined_books_js__syncBookHighlights(app, currentNote, bookId, { throwOnFail: true });
if (this._useLocalNoteContents) {
await this._inlined_amplenote_rw_js__flushLocalNotes(app);
}
if (success) {
await app.alert("✅ Book highlights fetched successfully!");
}
} catch (error) {
await app.alert(String(error));
}
},
/*******************************************************************************************/
async _syncOnly(app) {
try {
// As per docs: category is one of books, articles, tweets, supplementals or podcasts
const categories = ["books", "articles", "tweets", "supplementals", "podcasts"];
let result = await app.prompt(
"What category of highlights would you like to sync?", {
inputs: [{
type: "select",
label: "Category",
options: categories.map(function(value, index) {
return { value: value, label: value };
})
}]
}
);
if (result) await this._syncAll(app, result);
} catch (err) {
app.alert(err);
}
},
/*******************************************************************************************/
_initialize(app) {
this._abortExecution = false;
this._columnsBeforeUpdateDate = 6; // So, this is the 7th column in a 1-based table column array
this._columnsBeforeTitle = 1;
this._dateFormat = null;
this._forceReprocess = false;
this._lastRequestTime = null;
this._noteContents = {};
this._requestsCount = 0;
this._app = app;
// When doing mass updates, it's preferable to work with the string locally and replace the actual note content
// less often, since each note content replace triggers a redraw of the note table & all its per-row images.
// When this is enabled (globally, or for a particular method), the locally-manipulated note contents must be
// flushed to the actual note content via this._inlined_amplenote_rw_js__flushLocalNotes(app) before the method returns.
this._useLocalNoteContents = false;
if (this._testEnvironment === undefined) this._testEnvironment = false;
},
async _inlined_dashboard_js__writeDashboard(app, dashboard, dashboardNote) {
console.debug(`this._inlined_dashboard_js__writeDashboard()`);
// SORT each section
for (let [key, value] of Object.entries(dashboard)) {
dashboard[key] = value.sort(this._sortBooks);
}
// SORT the order of sections
dashboard = this._inlined_data_structures_js__distributeIntoSmallGroups(app, dashboard, this.constants.maxTableBooksPerSection);
let entries = Object.entries(dashboard);
entries.sort((a, b) => b[0].localeCompare(a[0]));
let dashboardMarkdown = this._inlined_markdown_js__markdownFromSections(app, entries, this._inlined_markdown_js__markdownFromTable.bind(this));
await this._inlined_amplenote_rw_js__replaceContent(dashboardNote, this.constants.dashboardBookListTitle, dashboardMarkdown);
},
async _inlined_dashboard_js__ensureBookInDashboardNoteTable(app, dashboard, bookObject) {
console.log(`this._inlined_dashboard_js__ensureBookInDashboardNoteTable(app, ${bookObject})`);
for (let year of Object.keys(dashboard)) {
let entries = dashboard[year];
for (let e of entries) {
console.debug(e["Book Title"]);
}
}
this._inlined_dashboard_js__removeBookFromDashboard(dashboard, bookObject);
let year = this._inlined_dashboard_js__sectionNameFromLastHighlight(bookObject.Updated);
if (year in dashboard) {
dashboard[year].push(bookObject);
dashboard[year] = dashboard[year].sort(this._sortBooks);
} else {
dashboard[year] = [bookObject];
}
},
_inlined_dashboard_js__sectionNameFromLastHighlight(lastHighlightDateString) {
let year = "";
if (lastHighlightDateString && this._inlined_dates_js__dateObjectFromDateString(lastHighlightDateString)) {
year = this._inlined_dates_js__dateObjectFromDateString(lastHighlightDateString).getFullYear();
} else {
year = this.constants.noHighlightSectionLabel;
}
return year;
},
_inlined_dashboard_js__removeBookFromDashboard(dashboard, bookObject) {
for (let year of Object.keys(dashboard)) {
const index = dashboard[year].findIndex(book => bookObject["Book Title"] === book["Book Title"]);
if (index !== -1) {
dashboard[year].splice(index, 1);
break;
}
}
},
async _inlined_dashboard_js__getReadwiseBookCount(app) {
const bookIndexResponse = await this._inlined_readwise_js__readwiseMakeRequest(app, `${ this.constants.readwiseBookIndexURL }?page_size=1`);
if (bookIndexResponse?.count) {
return bookIndexResponse.count;
}
else {
console.log("Did not received a Book index response from Readwise. Not updating Dashboard content");
return null;
}
},
async _inlined_dashboard_js__updateDashboardDetails(app, dashboard, details, {tableRowCount = null, bookCount = null } = {}) {
console.log(`this._inlined_dashboard_js__updateDashboardDetails(app, ${dashboard}, ${details}, ${tableRowCount}, ${bookCount} )`);
let dashDetails = this.constants.dashDetails;
const lastUpdatedAt = this._inlined_dashboard_js__boundaryBookUpdatedAtFromDashboard(dashboard, true);
const earliestUpdatedAt = this._inlined_dashboard_js__boundaryBookUpdatedAtFromDashboard(dashboard, false);
details[dashDetails.lastSyncedAt] = this._inlined_dates_js__localeDateFromIsoDate(app, new Date());
details[dashDetails.firstUpdated] = this._inlined_dates_js__localeDateFromIsoDate(app, earliestUpdatedAt);
details[dashDetails.lastUpdated] = this._inlined_dates_js__localeDateFromIsoDate(app, lastUpdatedAt);
details[dashDetails.booksImported] = tableRowCount;
let booksReported = details[dashDetails.booksReported];
details[dashDetails.booksReported] = bookCount ? bookCount : booksReported;
},
_inlined_dashboard_js__boundaryBookUpdatedAtFromDashboard(dashboard, findLatest) {
let result;
for (let group in dashboard) {
for (let item of dashboard[group]) {
let itemDate = this._inlined_dates_js__dateObjectFromDateString(item.Updated);
if (! itemDate || isNaN(itemDate.getTime())) {
// No usable date object from this row
} else if (!result || findLatest && itemDate > result || (!findLatest && itemDate < result)) {
result = itemDate;
}
}
}
console.debug("Found lastUpdatedAt", result, "aka", this._inlined_dates_js__localeDateFromIsoDate(result), "the", (findLatest ? "latest" : "earliest"), "record");
return result;
},
_inlined_dashboard_js__loadDetails(text) {
let lines = text.split('\n');
let details = {};
lines.forEach(line => {
if (!line.includes(":")) return;
let [key, value] = line.slice(2).split(': ');
// Try to convert string number to integer
let intValue = parseInt(value, 10);
details[key] = isNaN(intValue) ? value : intValue;
});
return details;
},
_inlined_dashboard_js__writeDetails(details) {
let text = '';
for (let key of Object.keys(details)) {
text += `- ${key}: ${details[key]}\n`;
}
return text;
},
async _inlined_dashboard_js__migrateBooksToSections(app, dashboardNote) {
console.log(`this._inlined_dashboard_js__migrateBooksToSections`);
const doMigrate = async () => {
const dashboardNoteContent = await this._inlined_amplenote_rw_js__noteContent(dashboardNote);
let dashboardBookListMarkdown = this._inlined_amplenote_rw_js__sectionContent(dashboardNoteContent, this.constants.dashboardBookListTitle);
let bookListRows = [];
if (dashboardBookListMarkdown) {
bookListRows = Array.from(dashboardBookListMarkdown.matchAll(/^(\|\s*![^\n]+)\n/gm));
if (bookListRows.length) {
console.debug("Found", bookListRows.length, "books to potentially migrate");
} else {
console.debug("No existing books found to migrate");
return;
}
} else {
console.debug("No dashboard book list found to migrate");
return;
}
const subSections = Array.from(dashboardBookListMarkdown.matchAll(/^##\s+([\w\s]+)/gm)).map(match =>
match[1].trim()).filter(w => w);
if (subSections.length && !subSections.find(heading => heading === this.constants.unsortedSectionTitle)) {
console.log("Book list is already in sections, no migration necessary");
return;
} else if (!dashboardBookListMarkdown.includes(this.constants.unsortedSectionTitle)) {
const unsortedSectionContent = `## ${ this.constants.unsortedSectionTitle }\n${ dashboardBookListMarkdown }`;
await this._inlined_amplenote_rw_js__replaceContent(dashboardNote, this.constants.dashboardBookListTitle, unsortedSectionContent);
// dashboardBookListMarkdown = this._inlined_amplenote_rw_js__sectionContent(await this._inlined_amplenote_rw_js__noteContent(dashboardNote), this.constants.dashboardBookListTitle);
console.log("Your Readwise library will be updated to split highlights into sections for faster future updates. This might take a few minutes if you have a large library.");
}
const dashboard = {};
const bookObjectList = this._inlined_markdown_js__tableFromMarkdown(dashboardBookListMarkdown);
const processed = [];
for (const bookObject of bookObjectList) {
console.debug("Processing", processed.length, "of", bookObjectList.length, "books");
await this._inlined_dashboard_js__ensureBookInDashboardNoteTable(app, dashboard, bookObject);
}
await this._inlined_dashboard_js__writeDashboard(app, dashboard, dashboardNote);
// Remove the old book list section
const unsortedContent = this._inlined_amplenote_rw_js__sectionContent(await this._inlined_amplenote_rw_js__noteContent(dashboardNote), this.constants.unsortedSectionTitle);
const unsortedWithoutTable = this._inlined_markdown_js__tableStrippedPreambleFromTable(unsortedContent);
if (unsortedContent.length && (unsortedWithoutTable?.trim()?.length || 0) === 0) {
await this._inlined_amplenote_rw_js__replaceContent(dashboardNote, this.constants.unsortedSectionTitle, "");
dashboardBookListMarkdown = this._inlined_amplenote_rw_js__sectionContent(await this._inlined_amplenote_rw_js__noteContent(dashboardNote), this.constants.dashboardBookListTitle);
try {
dashboardBookListMarkdown = dashboardBookListMarkdown.replace(new RegExp(`#+\\s${ this.constants.unsortedSectionTitle }[\\r\\n]*`), "");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message} (line 486)`));
}
}
await this._inlined_amplenote_rw_js__replaceContent(dashboardNote, this.constants.dashboardBookListTitle, dashboardBookListMarkdown.trim());
console.log("Successfully migrated books to yearly sections");
}
await this._inlined_amplenote_rw_js__flushLocalNotes(app);
};
await doMigrate();
},
_inlined_dashboard_js__sortBooks(a, b) {
// Sort highlights with missing date fields at the bottom
if (!a.Updated) {
if (a["Book Title"] < b["Book Title"]) return -1;
else return 1;
} else {
return new Date(b.Updated) - new Date(a.Updated);
}
},
_inlined_amplenote_rw_js__sectionContent(noteContent, headingTextOrSectionObject) {
console.debug(`this._inlined_amplenote_rw_js__sectionContent()`);
let sectionHeadingText;
if (typeof headingTextOrSectionObject === "string") {
sectionHeadingText = headingTextOrSectionObject;
} else {
sectionHeadingText = headingTextOrSectionObject.heading.text;
}
try {
sectionHeadingText = sectionHeadingText.replace(/^#+\s*/, "");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 1054)`));
}
}
const { startIndex, endIndex } = this._inlined_amplenote_rw_js__sectionRange(noteContent, sectionHeadingText);
return noteContent.slice(startIndex, endIndex);
},
async _inlined_amplenote_rw_js__replaceContent(note, sectionHeadingText, newContent, { level = 1 } = {}) {
console.log(`this._inlined_amplenote_rw_js__replaceContent() with this._useLocalNoteContents=${this._useLocalNoteContents}`);
const replaceTarget = this._inlined_markdown_js__sectionFromHeadingText(sectionHeadingText, { level });
if (this._useLocalNoteContents) {
let throughLevel = replaceTarget.heading?.level;
if (!throughLevel) throughLevel = sectionHeadingText.match(/^#*/)[0].length;
if (!throughLevel) throughLevel = 1;
const bodyContent = this._noteContents[note.uuid];
const { startIndex, endIndex } = this._inlined_amplenote_rw_js__sectionRange(bodyContent, sectionHeadingText);
if (startIndex) {
const revisedContent = `${ bodyContent.slice(0, startIndex) }\n${ newContent }${ bodyContent.slice(endIndex) }`;
this._noteContents[note.uuid] = revisedContent;
} else {
throw new Error(`Could not find section ${ sectionHeadingText } in note ${ note.name }`);
}
} else {
await note.replaceContent(newContent, replaceTarget);
}
},
_inlined_amplenote_rw_js__sectionRange(bodyContent, sectionHeadingText) {
console.debug(`this._inlined_amplenote_rw_js__sectionRange`);
const indexes = Array.from(bodyContent.matchAll(this.constants.sectionRegex));
const sectionMatch = indexes.find(m => m[1].trim() === sectionHeadingText.trim());
if (!sectionMatch) {
console.error("Could not find section", sectionHeadingText, "that was looked up. This might be expected");
return { startIndex: null, endIndex: null };
} else {
const level = sectionMatch[0].match(/^#+/)[0].length;
const nextMatch = indexes.find(m => m.index > sectionMatch.index && m[0].match(/^#+/)[0].length <= level);
const endIndex = nextMatch ? nextMatch.index : bodyContent.length;
return { startIndex: sectionMatch.index + sectionMatch[0].length + 1, endIndex };
}
},
async _inlined_amplenote_rw_js__insertContent(note, newContent, { atEnd = false } = {}) {
if (this._useLocalNoteContents) {
const oldContent = this._noteContents[note.uuid] || "";
if (atEnd) {
this._noteContents[note.uuid] = `${ oldContent.trim() }\n${ newContent }`;
} else {
this._noteContents[note.uuid] = `${ newContent.trim() }\n${ oldContent }`;
}
} else {
await note.insertContent(newContent, { atEnd });
}
},
async _inlined_amplenote_rw_js__sections(note, { minIndent = null } = {}) {
console.debug(`this._inlined_amplenote_rw_js__sections()`);
let sections;
if (this._useLocalNoteContents) {
const content = this._noteContents[note.uuid];
sections = this._inlined_markdown_js__getHeadingsFromMarkdown(content);
} else {
sections = await note.sections();
}
if (Number.isInteger(minIndent)) {
sections = sections.filter(section => (section.heading?.level >= minIndent) && section.heading.text.trim().length) || [];
return sections;
} else {
return sections;
}
},
async _inlined_amplenote_rw_js__noteContent(note) {
if (this._useLocalNoteContents) {
if (typeof this._noteContents[note.uuid] === "undefined") {
// Don't load too many notes in memory
if (Object.keys(this._noteContents).length >= this.constants.maxBookLimitInMemory) {
await this._inlined_amplenote_rw_js__flushLocalNotes(this._app);
}
this._noteContents[note.uuid] = await note.content();
}
return this._noteContents[note.uuid];
} else {
return await note.content();
}
},
async _inlined_amplenote_rw_js__flushLocalNotes(app) {
console.log("this._inlined_amplenote_rw_js__flushLocalNotes(app)");
for (const uuid in this._noteContents) {
console.log(`Flushing ${uuid}...`);
const note = await app.notes.find(uuid);
let content = this._noteContents[uuid];
if (!note.uuid.includes("local-")) {
// The note might be persisted to the sever, in which case its uuid changed
// In order to properly use the note later, we need to refer to it by its newer uuid
this._noteContents[note.uuid] = content;
}
let newContent = "";
// Replace note content with section names only, to avoid exceeding Amplenote write limit
// NOTE: this._useLocaLNoteContents has to be true here; might want to fix this dependency eventually
let sections = await this._inlined_amplenote_rw_js__sections(note);
console.debug(`Inserting sections ${sections.toString()}...`);
for (const section of sections) {
newContent = `${newContent}${this._inlined_markdown_js__mdSectionFromObject(section)}`;
}
await app.replaceNoteContent({uuid: note.uuid}, newContent);
// Replace individual sections with section content
sections = this._inlined_amplenote_rw_js__findLeafNodes(sections);
for (const section of sections) {
console.debug(`Inserting individual section content for ${section.heading.text}...`);
let newSectionContent = this._inlined_amplenote_rw_js__sectionContent(content, section);
await app.replaceNoteContent({uuid: note.uuid}, newSectionContent, {section});
}
delete this._noteContents[uuid];
delete this._noteContents[note.uuid];
}
},
_inlined_amplenote_rw_js__findLeafNodes(depths) {
let leafNodes = [];
for (let i = 0; i < depths.length - 1; i++) {
if (depths[i + 1].heading.level <= depths[i].heading.level) {
leafNodes.push(depths[i]);
}
}
// Add the last node if it's not already included (it's a leaf by default)
if (depths.length > 0 && (leafNodes.length === 0 || leafNodes[leafNodes.length - 1].heading.level !== depths.length - 1)) {
leafNodes.push(depths[depths.length - 1]);
}
return leafNodes;
},
_inlined_amplenote_rw_js__textToTagName(text) {
console.log("this._inlined_amplenote_rw_js__textToTagName", text);
if (!text) return null;
try {
return text.toLowerCase().trim().replace(/[^a-z0-9\/]/g, "-");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 1234)`));
}
}
},
_inlined_markdown_js__sectionFromHeadingText(headingText, { level = 1 } = {}) {
return { heading: { text: headingText, level }};
},
_inlined_markdown_js__markdownFromSections(app, sectionEntries, markdownFunction) {
let markdown = "";
for (let [key, value] of sectionEntries) {
markdown += `## ${ key }\n`;
markdown += markdownFunction(app, value);
}
return markdown;
},
_inlined_markdown_js__markdownFromHighlights(app, hls) {
let markdownLines = [];
for (let hl of hls) {
let result = "";
result += `> ### ${ hl.text }\n\n`;
// TODO: implement location
if (hl.note) result += `**Note**: ${ hl.note }\n`;
if (hl.color) result += `**Highlight color**: ${ hl.color }\n`;
result += `**Highlighted at**: ${ this._inlined_dates_js__localeDateFromIsoDate(app, hl.highlighted_at) } (#H${ hl.id })\n`;
markdownLines.push(result);
}
return markdownLines.join("\n\n");
},
_inlined_markdown_js__markdownFromTable(app, items) {
let headers = Object.keys(items[0]);
let markdown = "";
// Append table headers
markdown += this._inlined_markdown_js__tablePreambleFromHeaders(headers);
for (let item of items) {
markdown += this._inlined_markdown_js__markdownFromTableRow(headers, item);
}
markdown += '\n';
return markdown;
},
_inlined_markdown_js__tablePreambleFromHeaders(headers) {
let markdown = "";
markdown += `| ${ headers.map(item => `**${ item }**`).join(' | ') } |\n`;
markdown += `| ${ headers.map(() => '---').join(' | ') } |\n`;
return markdown;
},
_inlined_markdown_js__markdownFromTableRow(headers, item) {
let row;
try {
row = headers.map(header => item[header].replace(/(?, ",") || "");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 836)`));
}
}
let markdown = `| ${ row.join(' | ') } |\n`;
return markdown;
},
async _inlined_markdown_js__sectionsFromMarkdown(noteHandle, headingLabel, entriesFunction) {
console.debug(`this._inlined_markdown_js__sectionsFromMarkdown(noteHandle, ${ headingLabel }, entriesFunction)`);
const noteContent = await this._inlined_amplenote_rw_js__noteContent(noteHandle);
// This is the book list section
let mainSectionContent = this._inlined_amplenote_rw_js__sectionContent(noteContent, headingLabel);
// These will be year sections
let sections = this._inlined_markdown_js__getHeadingsFromMarkdown(mainSectionContent);
let result= [];
for (let section of sections) {
let yearMarkdownContent = this._inlined_amplenote_rw_js__sectionContent(mainSectionContent, section);
let entries = entriesFunction(yearMarkdownContent);
if (!entries) continue;
result = result.concat(entries);
}
return result;
},
_inlined_markdown_js__tableFromMarkdown(content) {
console.debug(`this._inlined_markdown_js__tableFromMarkdown(${content})`);
let lines = content.split('\n');
if (lines.length < 2) return null;
// Filter out any empty rows or rows that consist only of dashes or pipes
lines = lines.filter(row => row.trim() !== "" && !row.trim().match(/^\s*\|([-\s]+\|\s*)+$/));
let headers;
try {
headers = lines[0].split("|")
.slice(1, -1) // Remove first and last empty strings caused by the leading and trailing |
.map(header => header.trim().replace(new RegExp("\\*", "g"), ""));
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 887)`));
}
}
// Convert each row into a JavaScript object where each key is a header
// and each value is the corresponding cell in the row
const table = lines.slice(1).map(row => {
const cells = row.split(/(?)
.slice(1, -1) // Remove first and last empty strings caused by the leading and trailing |
.map(cell => cell.trim());
const rowObj = {};
headers.forEach((header, i) => {
rowObj[header] = cells[i] || null;
});
return rowObj;
});
return table;
},
_inlined_markdown_js__getHeadingsFromMarkdown(content) {
const headingMatches = Array.from(content.matchAll(/^#+\s*([^\n]+)/gm));
try {
return headingMatches.map(match => ({
heading: {
anchor: match[1].replace(/\s/g, "_"),
level: match[0].match(/^#+/)[0].length,
text: match[1],
}
}));
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 923)`));
}
}
},
_inlined_markdown_js__mdSectionFromObject(section) {
return `${"#".repeat(section.heading.level)} ${section.heading.text}\n`;
},
_inlined_markdown_js__tableStrippedPreambleFromTable(tableContent) {
try {
[
/^([|\s*]+(Cover|Book Title|Author|Category|Source|Highlights|Updated|Other Details)){1,10}[|\s*]*(?:[\r\n]+|$)/gm,
/^[|\-\s]+(?:[\r\n]+|$)/gm, // Remove top two rows that markdown tables export as of June 2023
].forEach(removeString => {
tableContent = tableContent.replace(removeString, "").trim();
tableContent = tableContent.replace(/^#+.*/g, ""); // Remove section label if present
});
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 949)`));
}
}
return tableContent;
},
_inlined_dates_js__yearFromDateString(dateString) {
const dateObject = this._inlined_dates_js__dateObjectFromDateString(dateString);
if (!dateObject) return null;
return dateObject.getFullYear();
},
_inlined_dates_js__localeDateFromIsoDate(app, dateStringOrObject) {
console.debug(`this._inlined_dates_js__localeDateFromIsoDate(app, ${dateStringOrObject}`);
try {
if (!dateStringOrObject) return "";
const dateObject = new Date(dateStringOrObject);
const dateFormat = this._dateFormat || (app && app.settings[this.constants.settingDateFormat]) || "en-US";
let result = dateObject.toLocaleDateString(dateFormat, { month: "long", day: "numeric", year: "numeric" });
const recentDateCutoff = (new Date()).setDate((new Date()).getDate() - 3);
if (dateObject > recentDateCutoff) {
result += " " + dateObject.toLocaleTimeString(dateFormat, { hour: "numeric", minute: "2-digit", hour12: true });
}
return result;
} catch (e) {
console.error("There was an error parsing your date string", dateStringOrObject, e);
return dateStringOrObject;
}
},
_inlined_dates_js__updateStampRegex() {
return new RegExp(`^(?:\\|[^|]*){${ this._columnsBeforeUpdateDate }}\\|\\s*([^|]+)\\s*\\|.*(?:$|[\\r\\n]+)`, "gm");
},
_inlined_dates_js__dateObjectFromDateString(dateString) {
console.log("this._inlined_dates_js__dateObjectFromDateString", dateString);
if (!dateString) return null;
let parseableString;
try {
parseableString = dateString.toLowerCase().replace(/\s?[ap]m/, "").replace(" at ", " ");
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 1293)`));
}
}
const parsedDate = Date.parse(parseableString);
if (parsedDate) {
return new Date(parsedDate);
} else {
return null;
}
},
async _inlined_dates_js__getLastUpdatedTimeFromNote(app, noteHandle) {
const content = await app.getNoteContent({ uuid: noteHandle.uuid });
if (!content) return null;
const lines = content.split("\n");
if (lines.length === 0) {
console.log("Found empty note parsing for date.");
return null;
}
// Translate our human friendly "June 6, 2023 at 5:01pm" into an object that Date.parse understands, e.g., June 6, 2023 17:01
const dateLine = lines.find(line => line.includes(this.constants.updateStringPreface));
let result = null;
if (dateLine) {
let dateString;
try {
dateString = dateLine.replace(this.constants.updateStringPreface, "");
if (dateString.includes("pm")) {
const hourMatch = dateString.match(/at\s([\d]{1,2}):/);
if (hourMatch) {
dateString = dateString.replace(` ${ hourMatch[1] }:`, ` ${ parseInt(hourMatch[1]) + 12 }:`);
} else {
console.error("Error parsing dateString");
}
}
} catch (err) {
if (err.name === "TypeError") {
throw(new Error(`${ err.message } (line 1335)`));
}
}
const result = this._inlined_dates_js__dateObjectFromDateString(dateString);
if (!result || isNaN(result.getTime())) {
console.log("Could not ascertain date from", dateLine, "and dateString", dateString);
return null;
}
} else {
console.log("Couldn't find a line containing the update time for note", noteHandle.uuid);
}
if (result) console.debug(`Last updated detected: ${ result.getTime() }`);
return result;
},
_inlined_data_structures_js__groupByValue(toGroup, groupFunction) {
let result = {};
for (let item of toGroup) {
let key = groupFunction(item);
if (key in result) {
result[key].push(item);
} else {
result[key] = [item];
}
}
return result;
},
_inlined_data_structures_js__distributeIntoSmallGroups(app, source, groupSize, trimFunction) {
let result = {};
for (let group of Object.keys(source)) {
let groupRows = [ ... source[group]];
let chunks = [];
while (groupRows.length) {
let toPush = groupRows.splice(0, groupSize);
if (trimFunction) {
toPush = trimFunction(app, toPush, groupSize);
}
chunks.push(toPush);
}
chunks.forEach((chunk, index) => {
result[`${ group }${ index > 0 ? ' (part ' + (index + 1) +')' : ''}`] = chunk;
});
}
return result;
},
_inlined_data_structures_js__trimHighlights(app, toPush) {
let groupMarkdown = this._inlined_markdown_js__markdownFromHighlights(app, toPush);
// Preview the markdown that will be generated; is it longer than 100k characters?
if (groupMarkdown.length < this.constants.maxReplaceContentLength) return toPush;
console.log("Trimming highlights for length...");
// When trimming highlights, trim only those exceeding 100.000 divided by how many
// highlights we have in the group
const itemTextLimit = this.constants.maxReplaceContentLength / toPush.length -
"> ### \n\n".length - "**Highlighted at**: \n".length - // Subtract the characters that we insert next to the highlight text itself
" (Trimmed for length)".length - // Subtract the text we add to explain a highlight was trimmed
30; // Subtract the length of the date string
for (const item of toPush) {
if (item.text.length > itemTextLimit) {
// Remove note and color fields from overflowing highlights
item.note = null;
item.color = null;
item.text = item.text.slice(0, itemTextLimit) + " (Trimmed for length)";
}
}
return toPush;
},
async *_inlined_readwise_js__testLongReadwiseFetchBooks(app, {bookIdFilter=null, categoryFilter=null, dateFilter=null} = {}) {
// TODO: add this as an automated test, please
let hls = [...Array(10).keys()];
hls = hls.map(item => ({
"id": item,
"text": "a".repeat(10000),
"location": 1,
"location_type": "order",
"note": null,
"color": "yellow",
"highlighted_at": "2022-09-13T16:41:53.186Z",
"created_at": "2022-09-13T16:41:53.186Z",
"updated_at": "2022-09-14T18:50:30.564Z",
"external_id": "6320b2bd7fbcdd7b0c000b3e",
"end_location": null,
"url": null,
"book_id": 123,
"tags": [],
"is_favorite": false,
"is_discard": false,
"readwise_url": "https://readwise.io/open/456"
}));
yield {
"user_book_id": 123,
"title": "Some title",
"author": "Some author",
"readable_title": "Some title",
"source": "raindrop",
"cover_image_url": "https://cover.com/image.png",
"unique_url": "",
"book_tags": [],
"category": "articles",
"document_note": "",
"readwise_url": "https://readwise.io/bookreview/123",
"source_url": "",
"asin": null,
"highlights": hls,
};
},
async *_inlined_readwise_js__readwiseFetchBooks(app, {bookIdFilter=null, categoryFilter=null, dateFilter=null} = {}) {
const url = new URL(`${ this.constants.readwiseExportURL }`);
if(bookIdFilter) url.searchParams.append("ids", bookIdFilter);
// Only apply date filters if we're fetching ALL types of books
if(dateFilter && !categoryFilter) url.searchParams.append("updatedAfter", dateFilter);
for await (const item of this._inlined_readwise_js__readwisePaginateExportRequest(app, url)) {
if (categoryFilter && item.category !== categoryFilter) continue;
yield item;
}
},
async *_inlined_readwise_js__readwisePaginateExportRequest(app, url) {
let nextPage = false;
while (true) {
if (nextPage) url.searchParams.set("pageCursor", nextPage);
const data = await this._inlined_readwise_js__readwiseMakeRequest(app, url);
if (data) {
for (const item of data.results) {
if (this._abortExecution) break;
// Update fields such because Readwise's EXPORT returns slightly different names than LIST
item.id = item.user_book_id;
item.num_highlights = item.highlights.length;
// Sort highlights by date descending
let hls = item.highlights;
hls = hls.sort((a, b) => {
// Sort highlights with missing date fields at the bottom
if (a.highlighted_at === undefined) return 1;
if (b.highlighted_at === undefined) return -1;
return new Date(b.highlighted_at) - new Date(a.highlighted_at);
});
item.highlights = hls;
item.last_highlight_at = null;
if (hls[0]) item.last_highlight_at = hls[0].highlighted_at;
yield item;
}
nextPage = data.nextPageCursor;
if (!nextPage) break;
} else {
console.error("Breaking from pagination loop due to no response from request", url);
break;
}
}
},
async *_inlined_readwise_js__readwiseGetAllHighlightsForBook(app, bookId, updatedAfter) {
console.log(`this._inlined_readwise_js__readwiseGetAllHighlightsForBook(app, ${ bookId }, ${ updatedAfter })`);
const url = new URL(`${ this.constants.readwiseHighlightsIndexURL }/`);
const params = new URLSearchParams();
params.append('book_id', bookId);
if (updatedAfter) {
params.append('updated__gt', updatedAfter.toISOString().slice(0, -1) + 'Z');
}
url.search = params;
yield* this._inlined_readwise_js__readwisePaginateRequest(app, url);
},
async *_inlined_readwise_js__readwisePaginateRequest(app, baseUrl) {
let currentPage = 1;
let hasNextPage = true;
while (hasNextPage) {
baseUrl.searchParams.append('page', currentPage);
baseUrl.searchParams.append('page_size', this.constants.readwisePageSize);
const data = await this._inlined_readwise_js__readwiseMakeRequest(app, baseUrl);
if (data) {
for (const item of data.results) {
if (this._abortExecution) break;
yield item;
}
hasNextPage = data.next !== null;
} else {
console.error("Breaking from pagination loop due to no response from request", baseUrl);
break;
}
currentPage++;
}
},
async _inlined_readwise_js__readwiseMakeRequest(app, url) {
console.log(`this._inlined_readwise_js__readwiseMakeRequest(app, ${url.toString()})`);
const readwiseAPIKey = app.settings["Readwise Access Token"];
if (!readwiseAPIKey || readwiseAPIKey.trim() === '') {
throw new Error('Readwise API key is empty. Please provide a valid API key.');
}
const headers = new Headers({ "Authorization": `Token ${ readwiseAPIKey }`, "Content-Type": 'application/json' });
// Wait to ensure we don't exceed the requests/minute quota of Readwise
await this._inlined_readwise_js__ensureRequestDelta(app);
// Use a proxy until Readwise adds CORS preflight headers
const proxyUrl = `https://amplenote-plugins-cors-anywhere.onrender.com/${ url.toString() }`;
const tryFetch = async () => {
const response = await fetch(proxyUrl, { method: 'GET', headers });
if (!response.ok) {
console.error(`HTTP error. Status: ${ response.status }`);
return null;
} else {
return response.json();
}
};
try {
let result = await tryFetch();
if (result) {
return result;
} else {
console.error("Null result trying fetch. Sleeping before final retry");
await new Promise(resolve => setTimeout(resolve,this.constants.sleepSecondsAfterRequestFail * 1000));
return await tryFetch();
}
} catch (e) {
console.trace();
console.error("Handling", e, "stack", e.stack);
app.alert("Error making request to Readwise", e);
return null;
}
},
async _inlined_readwise_js__ensureRequestDelta(app) {
const currentTime = new Date(); // Get the current time
if (this._lastRequestTime) { // Check if there was a previous request
const timeDifference = (currentTime - this._lastRequestTime) / 60000; // Calculate the time difference in minutes
if (timeDifference >= 1) {
this._requestsCount = 0; // Reset the request count if more than 1 minute has passed
}
// Check if the request count is greater than or equal to the rate limit
if (this._requestsCount >= this.constants.rateLimit) {
const waitTime = 60000 - timeDifference * 60000; // Calculate the remaining time in milliseconds before the next minute is reached
// Alert the user about the waiting time
const alertMessage = `Waiting for ${ Math.floor(waitTime / 1000) } seconds to satisfy Readwise API limit...\n\n` +
`You can wait here, or click "DONE" to dismiss, and we will update notes in the background as you work. ⏳\n\n` +
`Working concurrently while notes are being changed could lead to merge issues, so we recommend minimizing note changes while a sync is underway.`;
const response = await app.alert(alertMessage, { actions: [ { label: "Cancel sync", icon: "close" } ]});
if (response === 0) {
console.debug("User cancelled sync");
this._abortExecution = true;
} else {
await new Promise((resolve) => setTimeout(resolve, waitTime)); // Wait for the remaining time before making the next request
this._requestsCount = 0; // Reset the request count after waiting
}
}
}
this._lastRequestTime = currentTime; // Update the last request time to the current time
this._requestsCount++; // Increment the request count
},
async _inlined_books_js__syncBookHighlights(app, bookNote, readwiseBookID, { readwiseBook = null, throwOnFail = false } = {}) {
console.log(`this._inlined_books_js__syncBookHighlights(app, ${bookNote}, ${readwiseBookID})`);
let lastUpdatedAt = await this._inlined_dates_js__getLastUpdatedTimeFromNote(app, bookNote);
if (this._forceReprocess) {
lastUpdatedAt = null;
}
// Import all (new) highlights from the book
const noteContent = await this._inlined_amplenote_rw_js__noteContent(bookNote) || "";
if (!noteContent.includes("# Summary")) {
await this._inlined_amplenote_rw_js__insertContent(bookNote,"# Summary\n");
}
if (!readwiseBook) {
let generator = this._inlined_readwise_js__readwiseFetchBooks(app, {bookIdFilter: readwiseBookID});
let result = await generator.next();
readwiseBook = result.value;
if (!readwiseBook) {
if (throwOnFail) {
throw new Error(`Could not fetch book details for book ID ${ readwiseBookID }, you were probably rate-limited by Readwise. Please try again in 30-60 seconds?`);
} else {
return false;
}
}
}
const summaryContent = this._inlined_books_js__bookNotePrefaceContentFromReadwiseBook(app, readwiseBook, bookNote.uuid);
await this._inlined_amplenote_rw_js__replaceContent(bookNote, "Summary", summaryContent);
let highlightsContent = "";
if (!noteContent.includes("# Highlights")) {
await this._inlined_amplenote_rw_js__insertContent(bookNote, "\n# Highlights\n", { atEnd: true });
} else {
highlightsContent = this._inlined_amplenote_rw_js__sectionContent(noteContent, "Highlights");
}
let highlights = await this._inlined_markdown_js__sectionsFromMarkdown(bookNote, "Highlights", this._inlined_books_js__loadHighlights);
let bookNoteHighlightList = await this._inlined_books_js__bookHighlightsContentFromReadwiseBook(app, readwiseBook, highlights, lastUpdatedAt);
const sortOrder = app.settings[this.constants.settingSortOrderName] || this.constants.defaultHighlightSort;
let hlGroups = this._inlined_data_structures_js__groupByValue(bookNoteHighlightList,
item => {
if (!item.highlighted_at) return "No higlight date";
let year = this._inlined_dates_js__yearFromDateString(item.highlighted_at);
if (!year) return "No highlight date";
return year;
},
);
hlGroups = this._inlined_data_structures_js__distributeIntoSmallGroups(app, hlGroups, this.constants.maxBookHighlightsPerSection, this._inlined_data_structures_js__trimHighlights.bind(this));
let entries = Object.entries(hlGroups);
entries.sort((a, b) => b[0].localeCompare(a[0]));
let hlMarkdown = this._inlined_markdown_js__markdownFromSections(app, entries, this._inlined_markdown_js__markdownFromHighlights.bind(this));
try {
await this._inlined_amplenote_rw_js__replaceContent(bookNote, "Highlights", hlMarkdown);
} catch (error) {
console.log("Error replacing", readwiseBook.title, "content, length", hlMarkdown.length ," error", error);
}
let existingContent = "";
if (!noteContent.includes("Sync History")) {
await this._inlined_amplenote_rw_js__insertContent(bookNote, "\n# Sync History\n", { atEnd: true });
} else {
const match = noteContent.match(/#\sSync\sHistory\n([\s\S]+)$/m);
existingContent = match ? match[1] : "";
}
await this._inlined_amplenote_rw_js__replaceContent(bookNote, "Sync History",`${ this.constants.updateStringPreface }${ this._inlined_dates_js__localeDateFromIsoDate(app, new Date()) }\n` + existingContent);
return true;
},
async _inlined_books_js__ensureBookNote(app, readwiseBook) {
const baseTag = app.settings[this.constants.settingTagName] || this.constants.defaultBaseTag;
console.debug(`this._inlined_books_js__ensureBookNote(${ readwiseBook.title })`, baseTag);
// First, check if the note for this book exists
const readwiseNotes = await app.filterNotes({ tag: baseTag });
const bookRegex = new RegExp(`ID\\s?#${ readwiseBook.id }`);
const searchResults = readwiseNotes.filter(item => bookRegex.test(item.name));
let bookNote = null;
if (searchResults.length === 0) {
const noteTitle = this._inlined_books_js__noteTitleFromBook(readwiseBook);
// Create the note if it doesn't exist
const bookNoteTags = [`${ baseTag }/${ this._inlined_amplenote_rw_js__textToTagName(readwiseBook.category) }`];
if (app.settings[this.constants.settingAuthorTag] === "true") {
const candidateAuthorTag = this._inlined_amplenote_rw_js__textToTagName(readwiseBook.author);
const authorTag = candidateAuthorTag && candidateAuthorTag.split("-").slice(0, 3).join("-");
// Avoid inserting uber-long multi-author tag names that would pollute the tag space
if (authorTag) bookNoteTags.push(`${ baseTag }/${ authorTag }`);
}
bookNote = await app.notes.create(noteTitle, bookNoteTags);
} else {
const newNoteUUID = searchResults[0].uuid;
bookNote = await app.notes.find(newNoteUUID);
}
return bookNote;
},
_inlined_books_js__noteTitleFromBook(book) {
return `${ book.title } by ${ book.author } Highlights (ID #${ book.id })`;
},
_inlined_books_js__bookNotePrefaceContentFromReadwiseBook(app, book, bookNoteUUID) {
console.log("this._inlined_books_js__bookNotePrefaceContentFromReadwiseBook", JSON.stringify(book));
let sourceContent = book.source_url ? `[${ book.source }](${ book.source_url })` : book.source;
let asinContent = "";
if (book.asin) {
if (!book.source.toLowerCase()) console.error("Book ", book.title, "does not have a source?");
if (book.source?.toLowerCase()?.includes("kindle")) {
const kindleUrl = `kindle://book?action=open&asin=${ book.asin }`;
sourceContent = `[${ book.source }](${ kindleUrl })`;
asinContent = `ASIN: [${ book.asin }](${ kindleUrl })`;
} else {
asinContent = `ASIN: [${ book.asin }](https://www.amazon.com/dp/${ book.asin })`;
}
}
const baseTag = app.settings[this.constants.settingTagName] || this.constants.defaultBaseTag;
return `\n` +
`- **${ book.title }**\n` +
`- Book Author: [${ book.author }](/notes/${ bookNoteUUID }?tag=${ baseTag }/${ this._inlined_amplenote_rw_js__textToTagName(book.author) })\n` +
`- Category: ${ book.category }\n` +
`- Source: ${ sourceContent }\n` +
(asinContent ? `- ${ asinContent }\n` : "") +
`- Highlight count: ${ book.num_highlights }\n` +
`- Last highlight: ${ this._inlined_dates_js__localeDateFromIsoDate(app, book.last_highlight_at) }\n` +
`- [View all highlights on Readwise](https://readwise.io/bookreview/${ book.id })\n` +
`\n\n`; // Since replace will get rid of all content up to the next heading
},
_inlined_books_js__bookObjectFromReadwiseBook(app, readwiseBook, bookNoteUUID) {
console.debug(`this._inlined_books_js__bookObjectFromReadwiseBook(${readwiseBook})`);
let sourceContent = readwiseBook.source;
if (sourceContent === "kindle" && readwiseBook.asin) {
sourceContent = `[${ readwiseBook.source }](kindle://book?action=open&asin=${ readwiseBook.asin })`;
} else if (readwiseBook.source_url) {
sourceContent = `[${ readwiseBook.source }](${ readwiseBook.source_url })`;
}
return {
"Cover": `${ readwiseBook.cover_image_url ? `` : "[No cover image]" }`,
"Book Title": `[${ readwiseBook.title }](https://www.amplenote.com/notes/${ bookNoteUUID })`,
"Author": readwiseBook.author || "[No author]",
"Category": readwiseBook.category || "[No category]",
"Source": sourceContent || "[No source]",
"Highlights": `[${ readwiseBook.num_highlights } highlight${ readwiseBook.num_highlights === 1 ? "" : "s" }](https://www.amplenote.com/notes/${ bookNoteUUID }#Highlights})`,
"Updated": `${ readwiseBook.last_highlight_at ? this._inlined_dates_js__localeDateFromIsoDate(app, readwiseBook.last_highlight_at) : "No highlights" }`,
// `/bookreview/[\d]+` is used as a regex to grab Readwise book ID from row
"Other Details": `[Readwise link](https://readwise.io/bookreview/${ readwiseBook.id })`,
};
},
_inlined_books_js__loadHighlights(markdown) {
let result = [];
for (let hl of markdown.split("> ###")) {
let hlObject = {};
let lines = hl.split("\n");
hlObject.text = lines[0];
for (let i = 1; i < lines.length; i++) {
if (lines[i].startsWith('**Location**:')) {
hlObject.location = lines[i].substring(14);
} else if (lines[i].startsWith('**Highlighted at**:')) {
hlObject.highlighted_at = lines[i].substring(19);
} else if (lines[i].startsWith('**Note**:')) {
hlObject.note = lines[i].substring(9);
} else if (lines[i].startsWith('**Highlight color**:')) {
hlObject.color = lines[i].substring(20);
}
}
result.push(hlObject);
}
return result;
},
async _inlined_books_js__bookHighlightsContentFromReadwiseBook(app, readwiseBook, existingHighlights, lastUpdatedAt) {
console.log(`Getting all highlights for ${ readwiseBook.title }. Last updated at: ${ lastUpdatedAt }. Existing highlights length: ${ existingHighlights?.length }`);
// const newHighlightsList = _getNewHighlightsForBook(app, readwiseBook);
const newHighlightsList = readwiseBook.highlights;
let result = [];
if (newHighlightsList.length) {
// Highlight IDs were added June 2023, after initial launch. If the plugin content doesn't show existing signs of
// highlight IDs, we'll replace all highlight content to avoid dupes
let existingHighlightsContent = existingHighlights.join("\n");
if (/\(#H[\d]+\)/.test(existingHighlightsContent)) {
result = newHighlightsList.concat(existingHighlights);
} else {
result = newHighlightsList;
}
} else {
result = existingHighlights;
}
return result;
},
}