MM_33226 [v2] DataOperator - User section (#5255)

* MM_30475 : ADDED default schema

* MM_30475 : ADDED todo for field 'value' of default/Global entity

* MM_30476 : Created schema for SERVER DB

* MM_30476 : Server model [ IN PROGRESS ]

* MM_30476 : Including types for group, groups_in_channel and role

* MM_30476 : ADDED models for Group

- @typings absolute path has been added to the tsconfig.json

* MM_30476 : ADDED typings to current models

* MM_30476 : ADDED typings to current models

* MM_30476 : ADDED models related to TEAM section of the ERD

* MM_30476 : ADDED models for User section of the ERD

* MM_30476 : ADDED models for POST section of the ERD

* MM_30476 : ADDED models for Channel section of the ERD

* MM_30475 : Updated typings and references to MM_TABLES

* MM_30476 : Verified all field names

* MM_30476 : Verified every table associations

* MM_30476 : Verified all relation fields

* MM_30476 : Updated primary id of the main models

We will override the wdb id at component level when we create a new records.  This involves the models : channel, group, post, team and user.

* MM_30476 : Including 1:1 relationship amongs some entities

* MM_30476 : ADDED Schema Managers

* The migration array will hold all the migration steps.

*  The initial app release (e.g. v2 )will have an empty array and subsequent releases  (e.g.  v2.1 ) will have the steps listed in that array.

* On initialization, the database will perform the migration to accomodate for new columns/tables creation and while it will conserve the mobile phone's data, it will also make it conform to this new schema.

* If a migration fails, the migration process will rollback any changes.  This migration will be thoroughly tested in development before pushing it live.

* Revert "MM_30476 : ADDED Schema Managers"

This reverts commit a505bd5e11.

* MM_30478 : Converted schema_manager into a function

* MM_30478 : Updated schema manager and included patch for wdb

* MM_30478:  Updated watermelondb patch package

* MM_30478 : Update function create_schema_manager to createSqliteAdaptorOptions

* MM_30476 : Update constant name to reflect directory name

* MM_30476 : Updated msgCount from my_channel model to message_count in server schema

* MM_30482 : Added tests for schema_manager

* MM_30482 : Database Manager [ IN PROGRESS ]

* MM_30478 : Returning an sqliteAdapter instead of an object

* MM_30476 : Apply suggestions from code review

Co-authored-by: Elias Nahum <nahumhbl@gmail.com>

* MM_30476 : Updated all imports as per instruction.

* MM_30476 : Shortening object chains by destructuring

* MM_30476 : Updated schema file structure

* MM_30476 : Prettifying @typings folder

* MM_30476 : Removing useless ids

* MM_30476 : Prettify imports for decorators

* MM_30476 : ADDED documentations and lazy queries to Channel and Channel_Info

* MM_30476 : ADDED documentations for default schema

* MM_30476 : Documentation [ IN PROGRESS ]

- Following JSDoc syntax for single line comment
- Removed redundant fields in the 'membership' tables and left only the @relation records.

* MM_30476 : Documentations [ IN PROGRESS ]

* MM_30476 : Documentations [ IN PROGRESS ]

* MM_30476 : Documentations [ IN PROGRESS ]

* MM_30476 : Documentations [ IN PROGRESS]

Updated
1)  my_team and team,
2) my_channel and  channel,

to each have 1:1 relationship  with one another

* MM_30476 : Updated all Typescript definitions

* MM_30476 :Updated @relation to @immutableRelation

* MM_30476 : Updated description for previous_post_id

* MM_30478 : Updated patch package for wdb module

* MM_30478: DB Manager [IN PROGRESS ]

* MM_30478: DB Manager [IN PROGRESS]

* MM_30478: DB Manager [IN PROGRESS]

* MM_30478 : DB Manager [IN PROGRESS]

* MM_30478 : Deleting .db file on iOS

* MM_30478: Successfully deleting .db files and directory on iOS side

* MM_30478 : Update definition for default/global

* MM_30478 : Updated all models

* MM_30478 : Doing a bit of house cleaning

* MM_30478: Record of new server connection  added to default/servers db

* TS Definitely Typed Assignment issue is now FIXED

* MM_30478 : TS Definitely Typed Assignment \n  Removed all the constructors but error still in editor tabs.  But this time the app is not crashing

* MM_30478 : Attempt 1 [SUCCESSFUL]

* MM_30478 : Removing useDefineForClassFields

* MM_30478 : Retrieving the servers in a list + Improved the DB Manager and Babel config

* MM_30478 : Updated babel.config.js

* MM_30478 : Minor UI correction

* MM_30478 : Jest and Typescript configuration

* MM_30478 : A bit of housekeeping

* MM_30478 : Installed WDB on Android

* MM_30478 : Deletes new server record from default DB

* MM_30478 : Returns subset of server db instances

* MM_30478 : Code clean up

* MM_30478 :  Code clean up on db manager

* MM_30478 : House keeping + Patch for WDB

* MM_30478 : Android - Saving & Deleting in FilesDir [COMPLETED]

* MM_30478 : Code clean up

* MM_30478 : Code clean up

* MM_30478 : Code clean up

* MM_30478 : Test successful on Android device

* MM_30478 : Rolling back change to jest.config.js

* MM_30478 : Updated test to test_integration

* MM_30478 : Fix imports

* MM_30478 : Refactored the manual testscript

* MM_30478 : Renamed database manager test file

* MM_30478 : Code clean up

* MM_30478 : Updated manual test file with a note.

* MM_30482 : DataOperator [ IN PROGRESS ]

* MM_30482 : DataOperator - setting up the factory [ IN PROGRESS ]

* MM_30482: Code refactoring

* MM_30482 : DataOperator - setting up the factory [ IN PROGRESS ]

* MM_30482 : DataOperator - code clean up [ IN PROGRESS ]

* MM_30482 : Minor code clean up

* MM_30478 : Fixed JEST issue with TS

* MM_30478 : Fixed JEST issue with TS

* MM_30478 : Fixed JEST issue with TS

* MM_30478 : Implementing JEST test cases

* MM_30478 : Implementing JEST last  test cases

* MM_30478 : Jest fixing ts errors

* MM_30478 : Database Manager Jest testing [ IN PROGRESS ]

* MM_30482 - Fixing DataOperator [ IN PROGRESS ]

* MM_30482 : Code clean up

* MM_30482 - Creates multiple records [ IN PROGRESS ]

* MM_30482 - Creates multiple records [ IN PROGRESS ]

* MM_30482 : Update operation [ COMPLETED ]

* MM_30482 : Code clean up

* MM_30482 : Updated TS for Data Operator

* Update mobile v2 detox deps

* MM_30482 : Added factories for all isolated tables

* MM_30482 : Refactored TS

* MM_30482 : Refactored base factory

* MM_30482 : Updated JSDoc for operateBaseRecord - Delete CASE

* MM_30482 : Implementing test for Data Operator

* MM_30482 : Completed tests for all isolated tables

* MM_30482 : Renamed entity_factory into operators

* MM_30482 : Fix all imports

* MM_30482 : Update multiple records

* MM_30482 : Edge case for existing records ( update instead of create )

* MM_30482 : Edge case  - create instead of update

* MM_30482 : Code clean up

* MM_30482 : Code clean up

* MM_30482 : Code clean up

* MM_30482 : Code clean up

* Update app/database/admin/data_operator/operators.ts

Co-authored-by: Joseph Baylon <joseph.baylon@mattermost.com>

* Update app/database/admin/data_operator/operators.ts

Co-authored-by: Joseph Baylon <joseph.baylon@mattermost.com>

* Update app/database/admin/data_operator/operators.ts

Co-authored-by: Joseph Baylon <joseph.baylon@mattermost.com>

* MM_30482 : Imposing usage of correct table name for isolated entities

* MM_30482 : Code improvement as per Joseph reviews

* MM_30482 : Updated tests to validate choice of operator service wrt tableName

* MM_30482 : Updated PR as per suggestions

* MM_30482 : Updated comments to follow jsdoc conventions

* MM_33223 : Renamed DBInstance to DatabaseInstance

* MM_33223 : ADDED Prettier

* MM_33223 - Prettier formatting

* MM_33223 : Prettier formatting

* MM_33223 - Post section [ in progress ]

* MM_33223 : PostsInThread [99% completed ]

* MM_33223: Reaction entity completed

* MM_33223: Added Reaction to the Post

* MM_33223 : Refactored reactions utils

* MM_33223 : Added previous post id to all posts

* MM_33223 : Added File Metadata

* MM_33223 : Code clean up

* MM_33223 : Added PostMetadata

* MM_33223 : Added Draft

* MM_33223 - Removed Prettier

* MM_33223 - Undo files changes due to Prettier

* MM_33223 : Making use of MM eslint plugins

* MM_33223 : PostsInChannel [ IN PROGRESS ]

* MM_33223 : Including update_at in Post schema

* MM_33223: Code clean up

* MM_33223: Code clean up

* MM_33223 : Code clean up

* MM_33223: Testing Reaction [IN PROGRESS]

* MM_33223 : Updated typings for RawCustomEmoji in Reactions

* MM_33223 : Refactored DataOperator test

* MM_33223 : Jest - handleReactions - Completed

* MM_33223 : Jest - HandleDraft - Completed

* MM_33223 : Jest - HandleFiles - Completed

* MM_33223 : Refactored DataOperator-PostMetadata

* MM_33223 : Jest - HandlePostMetadata - Completed

* MM_33223 : Refactored posts into ordered and unordered

* MM_33223 : Refactoring + Jest Utils [ IN PROGRESS ]

* MM_33223 - Jest Utils - Completed

* MM_33223 : Jest - Remaining operators - Completed

* MM_33223 : Jest - Handler PostsInThread - Completed

* MM_33223 : Jest - HandlePostsInChannel - Completed

* MM_33223 : Refactored DataOperator class

* MM_33223 : DataOperator test clean up

* MM_33223 : DataOperator code clean up

* MM_33223 : Jest - HandlePosts - Completed

* MM_33223: JSDoc - Operators - Completed

* MM_33223 : Refactoring file types.ts

* MM_33223 : Refactored import statements

* MM_33223 : Added @database alias

* MM_33223 : Added missing JSDoc

* MM_33223 : Minor code clean up

* MM_33223 : Lint fixed

* MM_33223 : Disable eslint rules for Notification

* MM_33223 : Disable eslint rule for screens

* Update app/database/admin/data_operator/index.ts

Co-authored-by: Miguel Alatzar <migbot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Miguel Alatzar <migbot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Miguel Alatzar <migbot@users.noreply.github.com>

* MM_33223 : Update data_operatator as per suggestion

* MM_33226: Removed optType

* MM_33226 : ADDED User Handler+Operator+Jest, Included update_at field for User

* MM_33226 :  Preference entity - Completed

* MM_33226 : Team Membership entity - Completed

* MM_33226 : Team Membership - Jest - Completed

* MM_33226 : Removing duplicates for TeamMembership and Preferences

* MM_33226 : Refactored Custom Emojis to remove duplicates

* MM_33226 : Group Membership - Completed

* MM_33226 :  ChannelMembership - Completed

* MM_33226 : Refactored some handlers whose response have no Ids

* MM_33226 : Refactoring - in progress

* MM_33226 : Refactoring - in progress

* MM_33226 : Refactoring - in progress

* MM_33226 : Code clean up

* MM_33226 : Polishing Operator tests

* MM_33226 : Removing redundant test cases

* MM_33226 : Polishing Operators

* MM_33226 : Testing for duplicate post id in Raw  values

* MM_33226 : Including some error-throwing in the Database Manager

* MM_33226 : Merged in DataOperator/Post-section

* MM_33226 : Fixing the merging issues

* MM_33226 : fixing merge issues

* MM_33226 : Code polishing

* MM_33226 : Enabling user notify props comment

* MM_33226 : Correcting type casting

* MM_33226 : Correcting data operators

* MM_33226 : Corrections

* MM_33226 : Code clean up

* MM_33226 : Rename oneOfField to fieldName

* MM_33226 : Renaming comparators to Boolean name and oneOfField to fieldName

* MM_33226 : Putting back custom emoji into handleIsolatedEntity

* MM_33226 : Comparing simple arrays

* MM_33226 : Renaming DiscardDuplicates to ProcessInputs

* MM_33226 : Sort imports

* MM_33226 : Types clean up

Co-authored-by: Elias Nahum <nahumhbl@gmail.com>
Co-authored-by: Avinash Lingaloo <>
Co-authored-by: Joseph Baylon <joseph.baylon@mattermost.com>
Co-authored-by: Miguel Alatzar <migbot@users.noreply.github.com>
This commit is contained in:
Avinash Lingaloo
2021-04-02 00:26:05 +04:00
committed by GitHub
parent 040bf22264
commit e3f7c178bb
21 changed files with 3078 additions and 2040 deletions

View File

@@ -2,21 +2,36 @@
// See LICENSE.txt for license information.
import {Q} from '@nozbe/watermelondb';
import Model from '@nozbe/watermelondb/Model';
import {MM_TABLES} from '@constants/database';
import {ChainPosts, SanitizePosts, SanitizeReactions, RawPost, RawReaction} from '@typings/database/database';
import {
ChainPostsArgs,
IdenticalRecordArgs,
MatchExistingRecord,
RangeOfValueArgs,
RawPost,
RawReaction,
RawUser, RawValue,
RecordPair,
RetrieveRecordsArgs,
SanitizePostsArgs,
SanitizeReactionsArgs,
} from '@typings/database/database';
import Reaction from '@typings/database/reaction';
import Post from '@typings/database/post';
import {User} from '@database/server/models';
const {REACTION} = MM_TABLES.SERVER;
const {POST, USER, REACTION} = MM_TABLES.SERVER;
/**
* sanitizePosts: Creates arrays of ordered and unordered posts. Unordered posts are those posts that are not
* present in the orders array
* @param {SanitizePosts} sanitizePosts
* @param {SanitizePostsArgs} sanitizePosts
* @param {RawPost[]} sanitizePosts.posts
* @param {string[]} sanitizePosts.orders
*/
export const sanitizePosts = ({posts, orders}: SanitizePosts) => {
export const sanitizePosts = ({posts, orders}: SanitizePostsArgs) => {
const orderedPosts:RawPost[] = [];
const unOrderedPosts:RawPost[] = [];
@@ -29,22 +44,23 @@ export const sanitizePosts = ({posts, orders}: SanitizePosts) => {
});
return {
orderedPosts,
unOrderedPosts,
postsOrdered: orderedPosts,
postsUnordered: unOrderedPosts,
};
};
/**
* createPostsChain: Basically creates the 'chain of posts' using the 'orders' array; each post is linked to the other
* by the previous_post_id field.
* @param {ChainPosts} chainPosts
* @param {ChainPostsArgs} chainPosts
* @param {string[]} chainPosts.orders
* @param {RawPost[]} chainPosts.rawPosts
* @param {string} chainPosts.previousPostId
* @returns {RawPost[]}
*/
export const createPostsChain = ({orders, rawPosts, previousPostId = ''}: ChainPosts) => {
const posts: RawPost[] = [];
export const createPostsChain = ({orders, rawPosts, previousPostId = ''}: ChainPostsArgs) => {
const posts: MatchExistingRecord[] = [];
rawPosts.forEach((post) => {
const postId = post.id;
const orderIndex = orders.findIndex((order) => {
@@ -55,9 +71,9 @@ export const createPostsChain = ({orders, rawPosts, previousPostId = ''}: ChainP
// This case will not occur as we are using 'ordered' posts for this step. However, if this happens, that
// implies that we might be dealing with an unordered post and in which case we do not action on it.
} else if (orderIndex === 0) {
posts.push({...post, prev_post_id: previousPostId});
posts.push({record: undefined, raw: {...post, prev_post_id: previousPostId}});
} else {
posts.push({...post, prev_post_id: orders[orderIndex - 1]});
posts.push({record: undefined, raw: {...post, prev_post_id: orders[orderIndex - 1]}});
}
});
@@ -68,13 +84,13 @@ export const createPostsChain = ({orders, rawPosts, previousPostId = ''}: ChainP
* sanitizeReactions: Treats reactions happening on a Post. For example, a user can add/remove an emoji. Hence, this function
* tell us which reactions to create/delete in the Reaction table and which custom-emoji to create in our database.
* For more information, please have a look at https://community.mattermost.com/core/pl/rq9e8jnonpyrmnyxpuzyc4d6ko
* @param {SanitizeReactions} sanitizeReactions
* @param {SanitizeReactionsArgs} sanitizeReactions
* @param {Database} sanitizeReactions.database
* @param {string} sanitizeReactions.post_id
* @param {RawReaction[]} sanitizeReactions.rawReactions
* @returns {Promise<{createReactions: RawReaction[], createEmojis: {name: string}[], deleteReactions: Reaction[]}>}
*/
export const sanitizeReactions = async ({database, post_id, rawReactions}: SanitizeReactions) => {
export const sanitizeReactions = async ({database, post_id, rawReactions}: SanitizeReactionsArgs) => {
const reactions = (await database.collections.
get(REACTION).
query(Q.where('post_id', post_id)).
@@ -83,7 +99,7 @@ export const sanitizeReactions = async ({database, post_id, rawReactions}: Sanit
// similarObjects: Contains objects that are in both the RawReaction array and in the Reaction entity
const similarObjects: Reaction[] = [];
const createReactions: RawReaction[] = [];
const createReactions: MatchExistingRecord[] = [];
const emojiSet = new Set();
@@ -100,7 +116,7 @@ export const sanitizeReactions = async ({database, post_id, rawReactions}: Sanit
if (idxPresent === -1) {
// So, we don't have a similar Reaction object. That one is new...so we'll create it
createReactions.push(rawReaction);
createReactions.push({record: undefined, raw: rawReaction});
// If that reaction is new, that implies that the emoji might also be new
emojiSet.add(rawReaction.emoji_name);
@@ -121,3 +137,66 @@ export const sanitizeReactions = async ({database, post_id, rawReactions}: Sanit
return {createReactions, createEmojis, deleteReactions};
};
/**
* retrieveRecords: Retrieves records from the database
* @param {RetrieveRecordsArgs} records
* @param {Database} records.database
* @param {string} records.tableName
* @param {any} records.condition
* @returns {Promise<Model[]>}
*/
export const retrieveRecords = async ({database, tableName, condition}: RetrieveRecordsArgs) => {
const records = (await database.collections.get(tableName).query(condition).fetch()) as Model[];
return records;
};
/**
* hasSimilarUpdateAt: Database Operations on some entities are expensive. As such, we would like to operate if and only if we are
* 100% sure that the records are actually different from what we already have in the database.
* @param {IdenticalRecordArgs} identicalRecord
* @param {string} identicalRecord.tableName
* @param {RecordValue} identicalRecord.newValue
* @param {Model} identicalRecord.existingRecord
* @returns {boolean}
*/
export const hasSimilarUpdateAt = ({tableName, newValue, existingRecord}: IdenticalRecordArgs) => {
const guardTables = [POST, USER];
if (guardTables.includes(tableName)) {
type Raw = RawPost | RawUser
type ExistingRecord = Post | User
return (newValue as Raw).update_at === (existingRecord as ExistingRecord).updateAt;
}
return false;
};
/**
* This method extracts one particular field 'fieldName' from the raw values and returns them as a string array
* @param {RangeOfValueArgs} range
* @param {string} range.fieldName
* @param {RawValue[]} range.raws
* @returns {string[]}
*/
export const getRangeOfValues = ({fieldName, raws}: RangeOfValueArgs) => {
return raws.reduce((oneOfs, current: RawValue) => {
const key = fieldName as keyof typeof current;
const value: string = current[key] as string;
if (value) {
oneOfs.push(value);
}
return oneOfs;
}, [] as string[]);
};
/**
* getRawRecordPairs: Utility method that maps over the raws array to create an array of RecordPair
* @param {any[]} raws
* @returns {{record: undefined, raw: any}[]}
*/
export const getRawRecordPairs = (raws: any[]): RecordPair[] => {
return raws.map((raw) => {
return {raw, record: undefined};
});
};

View File

@@ -1,23 +1,24 @@
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
// See LICENSE.txt for license information.
import DatabaseManager from '@database/admin/database_manager';
import DataOperator from '@database/admin/data_operator';
import DatabaseManager from '@database/admin/database_manager';
import {DatabaseType} from '@typings/database/enums';
import {RawPost} from '@typings/database/database';
import {createPostsChain, sanitizePosts, sanitizeReactions} from './index';
import {mockedPosts, mockedReactions} from './mock';
jest.mock('../../database_manager');
jest.mock('@database/admin/database_manager');
describe('DataOperator: Utils tests', () => {
it('=> sanitizePosts: should filter between ordered and unordered posts', () => {
const {orderedPosts, unOrderedPosts} = sanitizePosts({
const {postsOrdered, postsUnordered} = sanitizePosts({
posts: Object.values(mockedPosts.posts),
orders: mockedPosts.order,
});
expect(orderedPosts.length).toBe(4);
expect(unOrderedPosts.length).toBe(2);
expect(postsOrdered.length).toBe(4);
expect(postsUnordered.length).toBe(2);
});
it('=> createPostsChain: should link posts amongst each other based on order array', () => {
@@ -30,29 +31,37 @@ describe('DataOperator: Utils tests', () => {
// eslint-disable-next-line max-nested-callbacks
const post1 = chainedOfPosts.find((post) => {
return post.id === '8swgtrrdiff89jnsiwiip3y1eoe';
});
const p = post.raw as unknown as RawPost;
return p.id === '8swgtrrdiff89jnsiwiip3y1eoe';
})?.raw as unknown as RawPost;
expect(post1).toBeTruthy();
expect(post1!.prev_post_id).toBe(previousPostId);
expect(post1?.prev_post_id).toBe(previousPostId);
// eslint-disable-next-line max-nested-callbacks
const post2 = chainedOfPosts.find((post) => {
return post.id === '8fcnk3p1jt8mmkaprgajoxz115a';
});
const p = post.raw as unknown as RawPost;
return p.id === '8fcnk3p1jt8mmkaprgajoxz115a';
})?.raw as unknown as RawPost;
expect(post2).toBeTruthy();
expect(post2!.prev_post_id).toBe('8swgtrrdiff89jnsiwiip3y1eoe');
// eslint-disable-next-line max-nested-callbacks
const post3 = chainedOfPosts.find((post) => {
return post.id === '3y3w3a6gkbg73bnj3xund9o5ic';
});
const p = post.raw as unknown as RawPost;
return p.id === '3y3w3a6gkbg73bnj3xund9o5ic';
})?.raw as unknown as RawPost;
expect(post3).toBeTruthy();
expect(post3!.prev_post_id).toBe('8fcnk3p1jt8mmkaprgajoxz115a');
expect(post3?.prev_post_id).toBe('8fcnk3p1jt8mmkaprgajoxz115a');
// eslint-disable-next-line max-nested-callbacks
const post4 = chainedOfPosts.find((post) => {
return post.id === '4btbnmticjgw7ewd3qopmpiwqw';
});
const p = post.raw as unknown as RawPost;
return p.id === '4btbnmticjgw7ewd3qopmpiwqw';
})?.raw as unknown as RawPost;
expect(post4).toBeTruthy();
expect(post4!.prev_post_id).toBe('3y3w3a6gkbg73bnj3xund9o5ic');
});