Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HOLD for payment 2024-04-09] [$500] [MEDIUM] Tag - The list of tags consisting of numbers is not ordered in ascending order. #33650

Closed
6 tasks done
kbecciv opened this issue Dec 27, 2023 · 66 comments
Assignees
Labels
Awaiting Payment Auto-added when associated PR is deployed to production Bug Something is broken. Auto assigns a BugZero manager. Daily KSv2 Engineering External Added to denote the issue can be worked on by a contributor

Comments

@kbecciv
Copy link

kbecciv commented Dec 27, 2023

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!


Version Number: 1.4.17-1
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:

Action Performed:

Preconditions:

  1. Add the employee account to the collection policy
  2. Navigate to the workspace tag settings
  3. Enable the tag
  4. Upload the following tag list CSV (https://drive.google.com/file/d/1q4ebThnmUTvpWhw-FYDfUSCg1UDZfi34/view?usp=drive_link
    )
  5. Be sure to select that the first row includes headers and to select Name on the mapping

Steps:

  1. Open https://staging.new.expensify.com/
  2. Log in as the employee account in NewDot
  3. Navigate to the workspace chat of the collect policy
  4. Start a Manual IOU request on the workspace chat
  5. Enter an amount and continue
  6. Click on "Show More"
  7. Click on the tag field

Expected Result:

The list of tags consisting of numbers should be ordered in ascending order (1,2,3,4,5...).

Actual Result:

The list of tags consisting of numbers is not ordered in ascending order.

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

  • Android: Native
  • Android: mWeb Chrome
  • iOS: Native
  • iOS: mWeb Safari
  • MacOS: Chrome / Safari
  • MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

Bug6326715_1703679900523.Recording__991.mp4

View all open jobs on GitHub

Upwork Automation - Do Not Edit
  • Upwork Job URL: https://www.upwork.com/jobs/~01d24e74dadb891171
  • Upwork Job ID: 1740069574100635648
  • Last Price Increase: 2024-01-10
  • Automatic offers:
    • situchan | Reviewer | 0
    • wildan-m | Contributor | 0
@kbecciv kbecciv added External Added to denote the issue can be worked on by a contributor Daily KSv2 Bug Something is broken. Auto assigns a BugZero manager. labels Dec 27, 2023
Copy link

melvin-bot bot commented Dec 27, 2023

Job added to Upwork: https://www.upwork.com/jobs/~01d24e74dadb891171

@melvin-bot melvin-bot bot changed the title Tag - The list of tags consisting of numbers is not ordered in ascending order. [$500] Tag - The list of tags consisting of numbers is not ordered in ascending order. Dec 27, 2023
Copy link

melvin-bot bot commented Dec 27, 2023

Triggered auto assignment to @sonialiap (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

@melvin-bot melvin-bot bot added the Help Wanted Apply this label when an issue is open to proposals by contributors label Dec 27, 2023
Copy link

melvin-bot bot commented Dec 27, 2023

Bug0 Triage Checklist (Main S/O)

  • This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
  • This bug is not a duplicate report (check E/App issues and #expensify-bugs)
    • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
  • This bug is reproducible using the reproduction steps in the OP. S/O
    • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
    • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
  • This issue is filled out as thoroughly and clearly as possible
    • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
  • I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

Copy link

melvin-bot bot commented Dec 27, 2023

Triggered auto assignment to Contributor-plus team member for initial proposal review - @situchan (External)

@rrrshtt
Copy link

rrrshtt commented Dec 27, 2023

Proposal

Please re-state the problem that we are trying to solve in this issue.

Numbers sorted in alphabetical order as plain text in tags.

What is the root cause of that problem?

function sortTags(tags) {

This function "Sorts tags alphabetically by name."
This function is used in OptionsListUtils.getFilteredOptions()

What changes do you think we should make in order to solve the problem?

We can update it to sort numbers as numbers firstly, and then sort everything else alphabetically. (We need requirements what goes first, numbers or text).

@rayane-d
Copy link
Contributor

rayane-d commented Dec 28, 2023

Proposal

Please re-state the problem that we are trying to solve in this issue.

The list of tags consisting of numbers is not ordered in ascending order

What is the root cause of that problem?

/**
* Sorts tags alphabetically by name.
*
* @param {Object<String, {name: String, enabled: Boolean}>} tags
* @returns {Array<Object>}
*/
function sortTags(tags) {
const sortedTags = _.chain(tags)
.values()
.sortBy((tag) => tag.name)
.value();
return sortedTags;
}

The sortTags function incorrectly sorts numerical tags as strings, leading to a non-sequential order (e.g., 1, 10, 2).
The issue stems from using Lodash's _.sortBy, which sorts elements as strings by default, not accounting for numeric values.

What changes do you think we should make in order to solve the problem?

Modify the sortTags function to differentiate between numeric and alphanumeric tags. Implement a conditional check within the sorting logic to treat numeric tags as numbers, ensuring correct numerical order.
as described here, we should start with Numbers first, then letters (a to z), then words in other scripts (arabic, chinese, etc.), then symbols.

we can use localeCompare with numeric option and with adjustment to make symbols at the end instead of at the begining:

function sortTags(tags) {
    // Function to check if a string contains symbols
    function isSymbol(s) {
        const symbolsRegex = /^[^\p{L}\p{N}]+$/u; // Matches symbols
// reference: https://www.regular-expressions.info/unicode.html?ref=bluebirz.net
        return symbolsRegex.test(s);
    }

    const sortedTags = tags.sort((tagA, tagB) => {
        // Extract the name property of the tags
        const nameA = tagA.name;
        const nameB = tagB.name;

        // Check if both tag names are symbols
        const isSymbolA = isSymbol(nameA);
        const isSymbolB = isSymbol(nameB);

        // Compare symbols vs. non-symbols
        if (isSymbolA && !isSymbolB) {
            return 1; // Move symbols to the end
        } else if (!isSymbolA && isSymbolB) {
            return -1; // Move symbols to the end
        }

        // If both are symbols or both are non-symbols, use localeCompare for sorting
// reference: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare
        return nameA.localeCompare(nameB, 'en', { numeric: true });
    });

    return sortedTags;
}

// Example usage
// you can test it in https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare
const tags = [
    { name: '0' },
    { name: '1' },
    { name: '2' },
    { name: '3' },
    { name: '10' },
    { name: '20' },
    { name: 'a' },
    { name: 'b' },
    { name: 'c' },
    { name: 'a1' },
    { name: 'a20' },
    { name: 'b1' },
    { name: 'b10' },
    { name: '日本' },
    { name: '中国' },
    { name: '!' },
    { name: '@' },
    { name: '#' },
    { name: '$' },
    { name: '0a' },
    { name: '10bc' },
    { name: '20a' }
];
const sortedTags = sortTags(tags);
const tagNames = sortedTags.map(tag => tag.name);
console.log(tagNames);

Result:

Array ["0", "0a", "1", "2", "3", "10", "10bc", "20", "20a", "a", "a1", "a20", "b", "b1", "b10", "c", "中国", "日本", "!", "@", "#", "$"]

What alternative solutions did you explore? (Optional)

@melvin-bot melvin-bot bot added the Overdue label Jan 1, 2024
Copy link

melvin-bot bot commented Jan 1, 2024

@sonialiap, @situchan Huh... This is 4 days overdue. Who can take care of this?

1 similar comment
Copy link

melvin-bot bot commented Jan 2, 2024

@sonialiap, @situchan Huh... This is 4 days overdue. Who can take care of this?

Copy link

melvin-bot bot commented Jan 3, 2024

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

Copy link

melvin-bot bot commented Jan 4, 2024

@sonialiap, @situchan Still overdue 6 days?! Let's take care of this!

@sonialiap
Copy link
Contributor

@situchan what do you think of the above proposals?

@melvin-bot melvin-bot bot removed the Overdue label Jan 4, 2024
@suneox
Copy link
Contributor

suneox commented Jan 6, 2024

Proposal

Please re-state the problem that we are trying to solve in this issue.

Tag - The list of tags consisting of numbers is not ordered in ascending order.

What is the root cause of that problem?

When sorting the list of tags, we use default sortBy alphabetically so the list of tags consisting of numbers is not ordered in ascending order.

function sortTags(tags: Record<string, Tag> | Tag[]) {
let sortedTags;
if (Array.isArray(tags)) {
sortedTags = tags.sort((a, b) => a.name.localeCompare(b.name));
} else {
sortedTags = Object.values(tags).sort((a, b) => a.name.localeCompare(b.name));
}
return sortedTags;
}

What changes do you think we should make in order to solve the problem?

We have a new sorting rule
and base on this comment so we will change the sortTags function to

function sortTags(tags: Record<string, Tag> | Tag[]) {
    const getGroupType = (str: string) => {
        // Only Number
        // if (/^\d+$/.test(str)) {
        //     return 0;
        // }
        // start with a number
        if (/^\d/.test(str)) {
            return 1;
        } 
        // start with a letter
        if (/^[a-zA-Z]/.test(str)) {
            return 2;
        }
        // everything else 
        return 3; 
    };

    if (Array.isArray(tags)) {
        const sortedTags = tags.sort((a, b) => {
            const typeA = getGroupType(a.name);
            const typeB = getGroupType(b.name);

            if (typeA !== typeB) {
                // sort by type first
                return typeA - typeB; 
            }

            // sort by number value
            // if (typeA === 0) {
            //     return Number(a.name) - Number(b.name);
            // }

            // perform a regular string comparison
            return a.name.localeCompare(b.name);
        });
        return sortedTags;
    }
    return sortTags(Object.values(tags));
}

What alternative solutions did you explore? (Optional)

If we want the sort rule to look like the image attachment in the new rule comment we can update the function sortTags

function sortTags(tags: Record<string, Tag> | Tag[]) {
    let sortedTags;

    if (Array.isArray(tags)) {
        sortedTags = tags.sort((a, b) => {
            const charCodeA = a.name.charCodeAt(0);
            const charCodeB = b.name.charCodeAt(0);

            if (charCodeA !== charCodeB) {
                return charCodeA - charCodeB;
            } 
            return a.name.localeCompare(b.name);
            
        });
    } else {
        sortedTags = Object.values(tags).sort((a, b) => {
            const charCodeA = a.name.charCodeAt(0);
            const charCodeB = b.name.charCodeAt(0);
            if (charCodeA !== charCodeB) {
                return charCodeA - charCodeB;
            }
            return a.name.localeCompare(b.name);
        });
    }

    return sortedTags;
}

POC

Screen.Recording.2024-03-01.at.00.49.59.mov

@Tony-MK
Copy link
Contributor

Tony-MK commented Jan 7, 2024

Proposal

Please re-state the problem that we are trying to solve in this issue.

Tag - The list of tags consisting of numbers is not ordered in ascending order.

What is the root cause of that problem?

It is impossible to sort strings appropriately using the inbuilt sortBy function because numbers represented as strings will be handled as strings.

Especially, the strings with different lengths pose issues with the sortBy function.
For example: 10 < 2 is true in the sortBy function.

function sortTags(tags) {
const sortedTags = _.chain(tags)
.values()
.sortBy((tag) => tag.name)
.value();
return sortedTags;

What changes do you think we should make in order to solve the problem?

We should replace the function sortTags function with the one below.

function sortTags(tags){
// Get the smallest character in the first index of each tag
const char = tags.map((tag) => tag.name.at(0)).sort().at(0);

// Get the maximum number of characters a tag name can have
const nChars = tags.map((tag) => tag.name.length).sort().at(0);

// Input: ['010', '2', '1', '3', 'a', '10', '011', '0.211', 'aa', 'aaa', '.', ',',  '!', '!.gf', 'sd.@', '.d', 's'];
tags.sort((x, y) => {
    const xNaN = isNaN(x.name);
    const yNaN = isNaN(y.name);
        
    // Different types, the number is the priority.
    if (xNaN != yNaN) return xNaN ? 1 : -1;

    // Compute a larger number
    if (!(xNaN && yNaN)) return x <= y ? (x == y ? 0 : -1) : 1;
    
    return x.padStart(nChars, char).localeCompare(y.padStart(nChars, char)); 
});

}
// Output: ['!', '!.gf', ',', '.', '.d', '0.211', '010', '011', '1', '2', '3', '10', 'a', 'aa', 'aaa', 's', 'sd.@']

@melvin-bot melvin-bot bot added the Overdue label Jan 7, 2024
@situchan
Copy link
Contributor

situchan commented Jan 8, 2024

I am not sure this is bug.
Why should we treat tags as numbers?

@melvin-bot melvin-bot bot removed the Overdue label Jan 8, 2024
Copy link

melvin-bot bot commented Jan 10, 2024

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

@situchan
Copy link
Contributor

Trying to get engineer's thoughts if this is bug or expected 🎀 👀 🎀

Copy link

melvin-bot bot commented Jan 10, 2024

Triggered auto assignment to @madmax330, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

Copy link

melvin-bot bot commented Jan 10, 2024

@madmax330 @sonialiap @situchan this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

@situchan
Copy link
Contributor

All proposed solutions don't satisfy above case.

@wildan-m
Copy link
Contributor

wildan-m commented Mar 23, 2024

Proposal

Please re-state the problem that we are trying to solve in this issue.

The tag sort order in new dot is not consistent with oldDot.

What is the root cause of that problem?

We are using different sorting algorithms; while oldDot uses underscore _.sortBy, newDot uses native JavaScript .sort. I believe they behave differently in some cases.

What changes do you think we should make in order to solve the problem?

To ensure consistency, we can utilize the same _.sortBy function for the new dot.

function sortTags(tags: Record<string, Tag> | Tag[]) {
    const tagsArray = Array.isArray(tags) ? tags : Object.values(tags);
    const sortedTags = _.sortBy(tagsArray, (obj) => {
        return localeCompare(obj.name, obj.name);
      });
    return sortedTags;
}

We also need to implement this in WorkspaceTagsPage.tsx

                _.sortBy(Object.values(policyTagList.tags || []), (obj) => {
                  return localeCompare(obj.name, obj.name);
                });

Not sure if the tag is also required to detect locale or not, but we can omit localeCompare if necessary.

It can also be applied to other fields if necessary and belong to the scope of this issue.

Branch to test

What alternative solutions did you explore? (Optional)

Change oldDot to use native .sort instead of _.sortBy

@situchan
Copy link
Contributor

@wildan-m's proposal looks good to me.
We should use lodashSortBy and also add unit tests.
🎀 👀 🎀 C+ reviewed

Copy link

melvin-bot bot commented Mar 23, 2024

Current assignee @madmax330 is eligible for the choreEngineerContributorManagement assigner, not assigning anyone new.

@melvin-bot melvin-bot bot removed the Help Wanted Apply this label when an issue is open to proposals by contributors label Mar 25, 2024
Copy link

melvin-bot bot commented Mar 25, 2024

📣 @situchan 🎉 An offer has been automatically sent to your Upwork account for the Reviewer role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job

Copy link

melvin-bot bot commented Mar 25, 2024

📣 @wildan-m 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

@melvin-bot melvin-bot bot added Reviewing Has a PR in review Weekly KSv2 and removed Daily KSv2 labels Mar 25, 2024
@wildan-m
Copy link
Contributor

@situchan The PR is ready for review.

@melvin-bot melvin-bot bot added Weekly KSv2 Awaiting Payment Auto-added when associated PR is deployed to production and removed Weekly KSv2 labels Apr 2, 2024
@melvin-bot melvin-bot bot changed the title [$500] [MEDIUM] Tag - The list of tags consisting of numbers is not ordered in ascending order. [HOLD for payment 2024-04-09] [$500] [MEDIUM] Tag - The list of tags consisting of numbers is not ordered in ascending order. Apr 2, 2024
Copy link

melvin-bot bot commented Apr 2, 2024

Reviewing label has been removed, please complete the "BugZero Checklist".

@melvin-bot melvin-bot bot removed the Reviewing Has a PR in review label Apr 2, 2024
Copy link

melvin-bot bot commented Apr 2, 2024

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.58-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

If no regressions arise, payment will be issued on 2024-04-09. 🎊

For reference, here are some details about the assignees on this issue:

Copy link

melvin-bot bot commented Apr 2, 2024

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

  • [@madmax330] The PR that introduced the bug has been identified. Link to the PR:
  • [@madmax330] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
  • [@madmax330] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
  • [@wildan-m / @situchan] Determine if we should create a regression test for this bug.
  • [@wildan-m / @situchan] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
  • [@sonialiap] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

@melvin-bot melvin-bot bot added Daily KSv2 and removed Weekly KSv2 labels Apr 8, 2024
@sonialiap
Copy link
Contributor

Payment summary:

@wildan-m
Copy link
Contributor

wildan-m commented Apr 9, 2024

Thanks @sonialiap, usually C+ will suggest conducting the regression test and completing the checklist, Is that correct?. In my opinion, I don't think a regression test is necessary as we have already written unit tests for the sorting function, and the bug does not directly affect a specific flow.

@sonialiap
Copy link
Contributor

That's correct, not sure why melvin tagged both you and Situ. Thanks for your thoughts on the regression test. Sounds like we're good on that front ✔️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Awaiting Payment Auto-added when associated PR is deployed to production Bug Something is broken. Auto assigns a BugZero manager. Daily KSv2 Engineering External Added to denote the issue can be worked on by a contributor
Projects
No open projects
Development

No branches or pull requests

9 participants