Bug 202850

Summary: Regression(r250786) API Test TestWebKitAPI.WebKit.DefaultQuota is flaky
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Chris Dumez <cdumez>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, achristensen, ap, cdumez, commit-queue, esprehn+autocc, ews-watchlist, ggaren, jbedard, kondapallykalyan, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=202553
https://bugs.webkit.org/show_bug.cgi?id=202590
Attachments:
Description Flags
Patch none

Aakash Jain
Reported 2019-10-11 09:16:37 PDT
API Test TestWebKitAPI.WebKit.DefaultQuota is failing consistently on macOS Release build. e.g.: https://build.webkit.org/builders/Apple%20High%20Sierra%20Release%20WK2%20%28Tests%29/builds/13637/steps/run-api-tests/logs/stdio https://ews-build.webkit.org/#/builders/3/builds/10036
Attachments
Patch (4.29 KB, patch)
2019-10-11 16:23 PDT, Chris Dumez 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Aakash Jain
Comment 1 2019-10-11 09:17:55 PDT
This test was modified recently in https://trac.webkit.org/changeset/250728/webkit
Alex Christensen
Comment 2 2019-10-11 13:48:23 PDT
That was a week ago. Has it been failing consistently since then? A sampling of the latest API test failures shows it is not failing consistently.
Jonathan Bedard
Comment 3 2019-10-11 14:05:23 PDT
Pretty much always fails on OpenSource High Sierra: <https://results.webkit.org/?suite=api-tests&test=TestWebKitAPI.WebKit.DefaultQuota> (unfortunately, we just brought up the database with the Catalina testers, so we don't have more results....)
Aakash Jain
Comment 4 2019-10-11 15:00:56 PDT
Actually it's not consistently failing, it's very flaky. For e.g.: in https://ews-build.webkit.org/#/builders/3/builds/10056, the test failed in run-api-tests step, but in immediate retry (re-run-api-tests) it passed. Similar thing happened in: https://ews-build.webkit.org/#/builders/3/builds/10054 https://ews-build.webkit.org/#/builders/3/builds/10048 https://ews-build.webkit.org/#/builders/3/builds/10018 https://ews-build.webkit.org/#/builders/3/builds/10005 Also, in following patches, it was flaky in a manner, which caused EWS to wrongly blame the patch being tested: https://ews-build.webkit.org/#/builders/3/builds/10037 https://ews-build.webkit.org/#/builders/3/builds/10019 https://ews-build.webkit.org/#/builders/3/builds/10014 https://ews-build.webkit.org/#/builders/3/builds/10006
Aakash Jain
Comment 5 2019-10-11 15:06:02 PDT
The first failure seems to be in https://ews-build.webkit.org/#/builders/3/builds/9629 followed by https://ews-build.webkit.org/#/builders/3/builds/9643 and https://ews-build.webkit.org/#/builders/3/builds/9729. All these patches belongs to Bug 202590. Bug 202590 has red EWS bubble for api-mac, indicating this failure, and that patch was committed. Seems like that commit broke this test.
Chris Dumez
Comment 6 2019-10-11 15:12:02 PDT
(In reply to Aakash Jain from comment #5) > The first failure seems to be in > https://ews-build.webkit.org/#/builders/3/builds/9629 > followed by https://ews-build.webkit.org/#/builders/3/builds/9643 and > https://ews-build.webkit.org/#/builders/3/builds/9729. All these patches > belongs to Bug 202590. > > Bug 202590 has red EWS bubble for api-mac, indicating this failure, and that > patch was committed. Seems like that commit broke this test. I will take a look but Bug 202590 is a PageCache patch and should have nothing to do with quotas.
Chris Dumez
Comment 7 2019-10-11 15:33:46 PDT
The test passes for me locally.
Chris Dumez
Comment 8 2019-10-11 15:40:18 PDT
+Youenn who wrote this test because I doubt this can be related to my PageCache change. Looking at this, I have no idea how it could be related.
Chris Dumez
Comment 9 2019-10-11 15:50:20 PDT
I was able to reproduce the timeout one time. I'll try running the test in a loop for make it more reproducible.
Chris Dumez
Comment 10 2019-10-11 15:52:10 PDT
(In reply to Chris Dumez from comment #9) > I was able to reproduce the timeout one time. I'll try running the test in a > loop for make it more reproducible. So, then it reproduces, with some extra logging I see: doTest(10) BEGIN (0) doTest(10) END (0) doTest(10) BEGIN (1) doTest(10) END (1) doTest(10) BEGIN (2) doTest(10) END (2) doTest(10) BEGIN (3) doTest(10) END (3) doTest(10) BEGIN (4) doTest(10) END (4) doTest(10) BEGIN (5) doTest(10) END (5) doTest(10) BEGIN (6) doTest(10) END (6) doTest(10) BEGIN (7) So the 7 first calls to doTest() caused us to receive a message back from the page, but the 8th one did not.
Chris Dumez
Comment 11 2019-10-11 15:59:28 PDT
(In reply to Chris Dumez from comment #10) > (In reply to Chris Dumez from comment #9) > > I was able to reproduce the timeout one time. I'll try running the test in a > > loop for make it more reproducible. > > So, then it reproduces, with some extra logging I see: > doTest(10) BEGIN (0) > doTest(10) END (0) > doTest(10) BEGIN (1) > doTest(10) END (1) > doTest(10) BEGIN (2) > doTest(10) END (2) > doTest(10) BEGIN (3) > doTest(10) END (3) > doTest(10) BEGIN (4) > doTest(10) END (4) > doTest(10) BEGIN (5) > doTest(10) END (5) > doTest(10) BEGIN (6) > doTest(10) END (6) > doTest(10) BEGIN (7) > > > So the 7 first calls to doTest() caused us to receive a message back from > the page, but the 8th one did not. I am making good progress debugging this. I should be able to fix this shortly.
Chris Dumez
Comment 12 2019-10-11 16:23:07 PDT
Created attachment 380798 [details] Patch
WebKit Commit Bot
Comment 13 2019-10-11 17:12:09 PDT
Comment on attachment 380798 [details] Patch Clearing flags on attachment: 380798 Committed r251031: <https://trac.webkit.org/changeset/251031>
WebKit Commit Bot
Comment 14 2019-10-11 17:12:11 PDT
All reviewed patches have been landed. Closing bug.
Radar WebKit Bug Importer
Comment 15 2019-10-15 16:48:10 PDT
<rdar://problem/56313729>
Aakash Jain
Comment 16 2019-10-30 18:33:24 PDT
This test flaked again 2 days ago. In https://ews-build.webkit.org/#/builders/3/builds/11094, it timed out in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.
Note You need to log in before you can comment on or make changes to this bug.